Jul 12 16:24:59.935060 ip-10-0-134-123 systemd[1]: Starting Kubernetes Kubelet... Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.402223 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409177 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409190 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409194 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409199 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409202 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409205 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409208 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 16:25:00.440867 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409211 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409214 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409217 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409220 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409222 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409225 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409227 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409230 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409234 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409236 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409239 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409242 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409244 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409247 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409256 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409259 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409262 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409265 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409268 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409270 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409273 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 16:25:00.442115 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409275 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409278 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409280 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409283 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409286 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409289 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409292 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409294 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409297 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409299 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409302 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409305 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409307 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409310 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409312 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409315 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409317 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409320 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409322 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 16:25:00.442978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409325 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409327 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409331 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409335 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409338 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409341 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409343 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409347 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409350 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409354 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409357 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409360 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409362 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409365 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409368 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409371 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409374 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409376 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409379 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 16:25:00.444043 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409382 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409384 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409387 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409389 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409392 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409395 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409397 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409400 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409402 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409404 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409406 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409409 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409413 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409416 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409418 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409421 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409423 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409426 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409428 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409431 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 16:25:00.444908 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409434 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409438 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409442 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409446 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409448 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409451 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409454 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409456 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409459 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409461 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409464 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409466 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.409470 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412227 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412241 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412245 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412248 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412251 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412254 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 16:25:00.445747 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412257 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412260 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412263 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412265 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412268 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412271 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412274 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412276 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412279 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412282 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412285 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412292 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412295 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412297 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412300 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412303 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412305 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412308 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412313 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 16:25:00.446611 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412316 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412320 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412323 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412326 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412329 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412333 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412336 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412339 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412343 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412347 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412350 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412353 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412356 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412359 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412361 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412365 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412367 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412370 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412372 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412375 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 16:25:00.447493 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412377 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412380 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412382 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412385 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412388 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412391 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412393 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412396 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412398 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412401 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412403 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412407 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412409 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412412 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412414 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412416 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412419 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412422 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412424 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 16:25:00.448385 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412427 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412430 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412432 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412435 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412439 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412441 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412444 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412446 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412449 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412451 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412454 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412457 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412459 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412462 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412464 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412467 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412470 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412474 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412489 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412493 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 16:25:00.449246 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412496 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412499 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412502 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412505 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412508 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412510 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412513 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412516 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412519 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412522 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412524 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412527 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412530 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412532 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.412534 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413357 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413365 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413368 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413373 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413377 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 16:25:00.450114 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413380 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413383 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413386 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413390 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413392 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413395 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413398 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413401 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413403 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413407 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413410 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413413 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413415 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413418 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413420 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413423 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413426 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413428 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413431 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 16:25:00.450978 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413433 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413436 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413439 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413441 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413444 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413447 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413449 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413451 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413454 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413457 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413460 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413462 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413465 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413468 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413471 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413474 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413476 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413479 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413481 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413484 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 16:25:00.452055 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413487 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413489 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413492 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413494 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413497 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413501 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413504 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413513 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413520 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413523 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413526 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413528 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413531 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413534 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413536 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413539 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413541 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413544 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413546 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413549 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 16:25:00.452942 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413552 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413554 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413557 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413560 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413562 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413565 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413567 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413570 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413573 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413575 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413578 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413581 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413583 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413586 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413588 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413591 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413593 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413596 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413599 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413602 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 16:25:00.454013 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413606 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413609 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413612 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413614 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413617 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413620 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413623 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413625 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413628 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413632 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413636 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413639 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413642 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413645 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.413649 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413718 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413726 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413732 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413736 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413740 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 12 16:25:00.455085 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413744 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413748 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413752 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413755 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413758 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413761 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413764 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413768 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413770 2575 flags.go:64] FLAG: --cgroup-root="" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413789 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413792 2575 flags.go:64] FLAG: --client-ca-file="" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413796 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413800 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413808 2575 flags.go:64] FLAG: --cluster-domain="" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413811 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413815 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413818 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413821 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413825 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413829 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413832 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413835 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413838 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413841 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 12 16:25:00.455952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413844 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413847 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413850 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413854 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413857 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413860 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413863 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413866 2575 flags.go:64] FLAG: --enable-server="true" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413869 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413873 2575 flags.go:64] FLAG: --event-burst="100" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413876 2575 flags.go:64] FLAG: --event-qps="50" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413879 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413882 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413885 2575 flags.go:64] FLAG: --eviction-hard="" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413889 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413891 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413894 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413897 2575 flags.go:64] FLAG: --eviction-soft="" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413900 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413903 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413909 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413912 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413916 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413919 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413922 2575 flags.go:64] FLAG: --feature-gates="" Jul 12 16:25:00.459216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413926 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413929 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413932 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413935 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413938 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413941 2575 flags.go:64] FLAG: --help="false" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413944 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-123.ec2.internal" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413947 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413950 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413953 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413956 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413960 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413962 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413965 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413968 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413971 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413974 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413977 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413980 2575 flags.go:64] FLAG: --kube-reserved="" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413983 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413985 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413988 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413991 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413994 2575 flags.go:64] FLAG: --lock-file="" Jul 12 16:25:00.460314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413997 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.413999 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414002 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414009 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414012 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414016 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414019 2575 flags.go:64] FLAG: --logging-format="text" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414022 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414025 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414028 2575 flags.go:64] FLAG: --manifest-url="" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414031 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414035 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414038 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414042 2575 flags.go:64] FLAG: --max-pods="110" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414045 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414048 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414051 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414054 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414057 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414060 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414063 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414071 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414074 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414077 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 12 16:25:01.865450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414080 2575 flags.go:64] FLAG: --pod-cidr="" Jul 12 16:25:00.501279 ip-10-0-134-123 systemd[1]: Started Kubernetes Kubelet. Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414083 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414089 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414091 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414094 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414098 2575 flags.go:64] FLAG: --port="10250" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414101 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414104 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b80ef4c380359e18" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414107 2575 flags.go:64] FLAG: --qos-reserved="" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414110 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414113 2575 flags.go:64] FLAG: --register-node="true" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414116 2575 flags.go:64] FLAG: --register-with-taints="" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414121 2575 flags.go:64] FLAG: --registry-burst="10" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414126 2575 flags.go:64] FLAG: --registry-qps="5" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414129 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414132 2575 flags.go:64] FLAG: --reserved-memory="" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414136 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414139 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414142 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414145 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414148 2575 flags.go:64] FLAG: --runonce="false" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414150 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414153 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414156 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414159 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 12 16:25:01.879147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414163 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414166 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414169 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414172 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414175 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414178 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414181 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414184 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414187 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414190 2575 flags.go:64] FLAG: --system-cgroups="" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414192 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414198 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414201 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414204 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414208 2575 flags.go:64] FLAG: --tls-min-version="" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414211 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414214 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414217 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414220 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414224 2575 flags.go:64] FLAG: --v="2" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414230 2575 flags.go:64] FLAG: --version="false" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414234 2575 flags.go:64] FLAG: --vmodule="" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414238 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.414241 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414355 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 16:25:01.881017 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414359 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414362 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414365 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414367 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414370 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414373 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414376 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414379 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414381 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414384 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414386 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414389 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414392 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414395 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414397 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414400 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414403 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414405 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414408 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414410 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 16:25:01.881916 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414413 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414415 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414418 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414420 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414423 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414425 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414429 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414433 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414436 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414439 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414441 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414445 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414449 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414452 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414455 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414458 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414460 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414463 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414465 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 16:25:01.882539 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414468 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414471 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414473 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414476 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414479 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414481 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414484 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414486 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414489 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414492 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414494 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414497 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414499 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414502 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414504 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414507 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414509 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414512 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414514 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414518 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 16:25:01.883152 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414522 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414525 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414528 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414530 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414533 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414537 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414540 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414543 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414546 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414550 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414554 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414557 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414560 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414563 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414566 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414569 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414571 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414574 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414577 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 16:25:01.883783 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414579 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414582 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414585 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414587 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414590 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414593 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414595 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414597 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414600 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414602 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414605 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414607 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414611 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414615 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414618 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414620 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414623 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414626 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414628 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 16:25:01.884406 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.414631 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.415255 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.422012 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.422029 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.422044 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.422050 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422105 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422109 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422113 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422116 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422120 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422122 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422125 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422128 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422131 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 16:25:01.884953 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422134 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422137 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422141 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422146 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422149 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422152 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422156 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422158 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422161 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422165 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422168 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422171 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422173 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422176 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422178 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422181 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422184 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422188 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422191 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 16:25:01.885536 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422194 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422213 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422219 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422222 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422224 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422227 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422230 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422234 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422237 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422241 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422244 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422247 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422250 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422253 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422257 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422260 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422263 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422266 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422269 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 16:25:01.886097 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422272 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422275 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422278 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422281 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422287 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422289 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422292 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422294 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422297 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422299 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422302 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422305 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422307 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422310 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422313 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422317 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422320 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422323 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422325 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422328 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 16:25:01.886789 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422331 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422333 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422336 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422338 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422341 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422344 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422346 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422349 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422352 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422354 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422357 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422360 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422362 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422365 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422367 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422370 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422374 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422376 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422379 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422381 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 16:25:01.887480 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422384 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422387 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422389 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422392 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422394 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422397 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422399 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422403 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422406 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422409 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422412 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422414 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.422420 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422532 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422537 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 16:25:01.888071 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422540 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422543 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422545 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422548 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422550 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422553 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422556 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422559 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422561 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422564 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422567 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422569 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422572 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422576 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422578 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422581 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422583 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422586 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422588 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422591 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 16:25:01.888540 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422593 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422595 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422598 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422600 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422603 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422607 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422610 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422612 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422615 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422618 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422620 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422623 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422626 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422628 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422631 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422633 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422636 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422638 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422641 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 16:25:01.889105 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422643 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422646 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422649 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422651 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422654 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422656 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422660 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422663 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422665 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422668 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422671 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422673 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422676 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422679 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422681 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422684 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422686 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422689 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422692 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422695 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 16:25:01.889646 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422698 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422701 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422703 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422706 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422708 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422711 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422713 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422716 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422718 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422722 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422724 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422727 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422729 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422732 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422736 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422740 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422743 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422746 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422750 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 16:25:01.890248 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422752 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422755 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422757 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422760 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422762 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422765 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422768 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422770 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422788 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422793 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422796 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422800 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422803 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422806 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422811 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422814 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422817 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422819 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 16:25:01.890753 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:00.422822 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.422827 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.422947 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.425723 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.426963 2575 server.go:1019] "Starting client certificate rotation" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.427060 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.427089 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.429497 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.433126 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.457918 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.465770 2575 log.go:25] "Validated CRI v1 runtime API" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.467889 2575 log.go:25] "Validated CRI v1 image API" Jul 12 16:25:01.891365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.471171 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 12 16:25:01.891999 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.474066 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8dfda6b3-5421-4b11-93bc-5dd5164cf605:/dev/nvme0n1p4 ec0ee305-608c-415c-b4ce-5001dc0e2681:/dev/nvme0n1p3] Jul 12 16:25:01.891999 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.474091 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 12 16:25:01.891999 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.478570 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mj7rg" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.479937 2575 manager.go:216] Machine: {Timestamp:2026-07-12 16:25:00.478610876 +0000 UTC m=+0.418452109 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099384 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa8014a1f91d468bc6979dfbf9374 SystemUUID:ec2aa801-4a1f-91d4-68bc-6979dfbf9374 BootID:0e5ea806-ddbb-4577-b720-345c23f0203f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:47:6f:67:5f:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:47:6f:67:5f:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:98:28:d4:dd:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.480052 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.480178 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.482996 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.483023 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.483656 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.483666 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.483674 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.484145 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.486068 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mj7rg" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.487195 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.490937 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.491047 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.491079 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.491089 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.491098 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 12 16:25:02.186180 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.491108 2575 kubelet.go:398] "Adding apiserver pod source" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.491118 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.491496 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.491513 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.493813 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.495573 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.495600 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.495748 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.495761 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.495767 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.495788 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.495799 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.495805 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.496432 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.496438 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.496445 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.496451 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.496460 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.496468 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.497200 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.497207 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.500321 2575 server.go:1268] "Started kubelet" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.500393 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.500414 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.500489 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.508257 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.508324 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.508981 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.510627 2575 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-123.ec2.internal" not found Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.510686 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:00.510719 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.512869 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.512881 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:00.512953 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-123.ec2.internal\" not found" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.513457 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.513469 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.513484 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.513563 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.513572 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.513805 2575 factory.go:55] Registering systemd factory Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.513816 2575 factory.go:223] Registration of the systemd container factory successfully Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.513989 2575 factory.go:153] Registering CRI-O factory Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.514003 2575 factory.go:223] Registration of the crio container factory successfully Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.514047 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.514071 2575 factory.go:103] Registering Raw factory Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.514089 2575 manager.go:1196] Started watching for new ooms in manager Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.514498 2575 manager.go:349] Starting recovery of all containers Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.514948 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:00.517914 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-123.ec2.internal\" not found" node="ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.526269 2575 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-123.ec2.internal" not found Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:00.528929 2575 manager.go:354] Recovery completed Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.704409 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 16:25:02.187157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.707028 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasSufficientMemory" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.707055 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.707069 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasSufficientPID" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.707659 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.707667 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.707667 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.707740 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.708317 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasSufficientMemory" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.708339 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.708348 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasSufficientPID" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.709097 2575 policy_none.go:49] "None policy: Start" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.709118 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.709128 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.710085 2575 policy_none.go:47] "Start" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.749798 2575 manager.go:337] "Starting Device Plugin manager" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:01.749993 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.750004 2575 server.go:87] "Starting device plugin registration server" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.750236 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.750254 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.750348 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.750425 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.750433 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:01.750987 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:01.751036 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-123.ec2.internal\" not found" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.754008 2575 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-123.ec2.internal" not found Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.850841 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.852339 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasSufficientMemory" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.852374 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.852392 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeHasSufficientPID" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.852433 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.861712 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:01.861732 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-123.ec2.internal\": node \"ip-10-0-134-123.ec2.internal\" not found" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.867046 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.868334 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.868383 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.868405 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:01.868444 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.870526 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.969540 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal"] Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.971055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.972058 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.986112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0e86ff064f57baa490d1d3b0d45fbf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal\" (UID: \"0e86ff064f57baa490d1d3b0d45fbf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.188618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.986139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e86ff064f57baa490d1d3b0d45fbf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal\" (UID: \"0e86ff064f57baa490d1d3b0d45fbf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.986159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b6203c45afbfc7cc0bd349ce0d548f2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-123.ec2.internal\" (UID: \"4b6203c45afbfc7cc0bd349ce0d548f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:01.999829 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.004218 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.015363 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.072782 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.087018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0e86ff064f57baa490d1d3b0d45fbf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal\" (UID: \"0e86ff064f57baa490d1d3b0d45fbf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.087048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e86ff064f57baa490d1d3b0d45fbf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal\" (UID: \"0e86ff064f57baa490d1d3b0d45fbf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.087069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b6203c45afbfc7cc0bd349ce0d548f2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-123.ec2.internal\" (UID: \"4b6203c45afbfc7cc0bd349ce0d548f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.087116 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b6203c45afbfc7cc0bd349ce0d548f2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-123.ec2.internal\" (UID: \"4b6203c45afbfc7cc0bd349ce0d548f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.087124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e86ff064f57baa490d1d3b0d45fbf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal\" (UID: \"0e86ff064f57baa490d1d3b0d45fbf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.189762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.087124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0e86ff064f57baa490d1d3b0d45fbf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal\" (UID: \"0e86ff064f57baa490d1d3b0d45fbf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.462074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.317609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.462074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.376285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.596444 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.596572 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.596599 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.596622 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.661543 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 16:20:01 +0000 UTC" deadline="2028-02-17 14:44:24.08893671 +0000 UTC" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.661583 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14038h19m21.427355896s" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.662620 2575 apiserver.go:52] "Watching apiserver" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.669480 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.669794 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-6vlfb","openshift-image-registry/node-ca-c52dt","openshift-network-operator/iptables-alerter-8js4s","openshift-ovn-kubernetes/ovnkube-node-zxb5k","kube-system/global-pull-secret-syncer-4z46t","kube-system/konnectivity-agent-5r44g","kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal","openshift-multus/multus-additional-cni-plugins-wjtqp","openshift-multus/multus-wx9bn","openshift-multus/network-metrics-daemon-hpv4n","openshift-network-diagnostics/network-check-target-blsjj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg","openshift-cluster-node-tuning-operator/tuned-plvmn"] Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.671155 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.672129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.672175 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.673176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.673399 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7qc49\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.673403 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.673405 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.674420 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.675390 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.675615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.675668 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.675669 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4lkgm\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.675971 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.676930 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j8tt6\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.677169 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.678140 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.678168 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.678175 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.678280 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.745562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.679026 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.679079 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.680203 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.681404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.681674 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.681872 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4fg8w\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.681922 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.682025 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.682062 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.682704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.683379 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.683643 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fxblm\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.683650 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.683647 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.684017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.684024 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.684076 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.684411 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.684601 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.685357 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.686589 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.686847 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h4xhp\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.686848 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687129 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687160 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687233 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687264 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9qnj\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687163 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687455 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5h4bs\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687517 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687672 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.687750 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6qrhb\"" Jul 12 16:25:02.746765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-kubelet\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cnibin\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-run-netns\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-system-cni-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-etc-kubernetes\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a6c1585-8073-46aa-a8d3-9de8a53868a9-host-slash\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovn-node-metrics-cert\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-conf-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99b9a17e-370f-43bc-8186-fe34fe8a47c6-agent-certs\") pod \"konnectivity-agent-5r44g\" (UID: \"99b9a17e-370f-43bc-8186-fe34fe8a47c6\") " pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.688986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-slash\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-env-overrides\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xptf7\" (UniqueName: \"kubernetes.io/projected/207a370a-c6ee-42b3-a858-1f9ec07a7191-kube-api-access-xptf7\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovnkube-config\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc62v\" (UniqueName: \"kubernetes.io/projected/934d23da-7dc2-40c1-84c1-9784f7a99dde-kube-api-access-fc62v\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689144 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-hosts-file\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:02.747764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-cnibin\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-socket-dir-parent\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689244 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-multus-certs\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689268 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8pzf\" (UniqueName: \"kubernetes.io/projected/0a6c1585-8073-46aa-a8d3-9de8a53868a9-kube-api-access-v8pzf\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7lk\" (UniqueName: \"kubernetes.io/projected/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-kube-api-access-zz7lk\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-netns\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-etc-selinux\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-kubelet\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-systemd-units\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-etc-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-dbus\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-tmp-dir\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd0c6f87-ae63-46bc-9373-da4088aeba89-host\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.748838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovnkube-script-lib\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-cni-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-cni-multus\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689665 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-cni-bin\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689718 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-os-release\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd0c6f87-ae63-46bc-9373-da4088aeba89-serviceca\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a6c1585-8073-46aa-a8d3-9de8a53868a9-iptables-alerter-script\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-log-socket\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-hostroot\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-ovn\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5993a850-124f-4910-9e5e-36e75c22589c-cni-binary-copy\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-device-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:02.749396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.689989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-node-log\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-k8s-cni-cncf-io\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjj8p\" (UniqueName: \"kubernetes.io/projected/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-kube-api-access-cjj8p\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690051 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4r4j\" (UniqueName: \"kubernetes.io/projected/dd0c6f87-ae63-46bc-9373-da4088aeba89-kube-api-access-t4r4j\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-sys-fs\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-kubelet-config\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-socket-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-run-ovn-kubernetes\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5993a850-124f-4910-9e5e-36e75c22589c-multus-daemon-config\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddl2z\" (UniqueName: \"kubernetes.io/projected/5993a850-124f-4910-9e5e-36e75c22589c-kube-api-access-ddl2z\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690222 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690240 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tttt\" (UniqueName: \"kubernetes.io/projected/798cf565-5257-4b14-8348-47dd6ed08a49-kube-api-access-9tttt\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-var-lib-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-cni-netd\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-system-cni-dir\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-registration-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:02.749946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99b9a17e-370f-43bc-8186-fe34fe8a47c6-konnectivity-ca\") pod \"konnectivity-agent-5r44g\" (UID: \"99b9a17e-370f-43bc-8186-fe34fe8a47c6\") " pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:02.750550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-os-release\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.750550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-cni-bin\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:02.750550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.690511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-systemd\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:02.750550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.702860 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 16:25:02.750550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.749787 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4mhg2" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.756715 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4mhg2" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.784735 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.790871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-hosts-file\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.790900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-cnibin\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.790917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-socket-dir-parent\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.790932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-multus-certs\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.790946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.790963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.790978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8pzf\" (UniqueName: \"kubernetes.io/projected/0a6c1585-8073-46aa-a8d3-9de8a53868a9-kube-api-access-v8pzf\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.790992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7lk\" (UniqueName: \"kubernetes.io/projected/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-kube-api-access-zz7lk\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-lib-modules\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791058 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-multus-certs\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-netns\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-etc-selinux\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-kubelet\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-systemd-units\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-etc-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-dbus\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:03.001397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-tmp-dir\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-tuned\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd0c6f87-ae63-46bc-9373-da4088aeba89-host\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovnkube-script-lib\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-host\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-cni-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-cni-multus\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-cni-bin\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-kubernetes\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysctl-conf\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-var-lib-kubelet\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-os-release\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791499 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.002401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd0c6f87-ae63-46bc-9373-da4088aeba89-serviceca\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791571 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-netns\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791656 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-etc-selinux\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-kubelet\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791696 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-systemd-units\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-etc-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791812 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd0c6f87-ae63-46bc-9373-da4088aeba89-serviceca\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.791815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-dbus\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-tmp-dir\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792045 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd0c6f87-ae63-46bc-9373-da4088aeba89-host\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a6c1585-8073-46aa-a8d3-9de8a53868a9-iptables-alerter-script\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-log-socket\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-hostroot\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-ovn\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792260 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-cnibin\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5993a850-124f-4910-9e5e-36e75c22589c-cni-binary-copy\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.003136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-hosts-file\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-device-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-socket-dir-parent\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-node-log\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysconfig\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-cni-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-systemd\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792479 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovnkube-script-lib\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-sys\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-device-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smbb\" (UniqueName: \"kubernetes.io/projected/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-kube-api-access-7smbb\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-os-release\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792585 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-cni-multus\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-k8s-cni-cncf-io\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjj8p\" (UniqueName: \"kubernetes.io/projected/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-kube-api-access-cjj8p\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-ovn\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.003891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4r4j\" (UniqueName: \"kubernetes.io/projected/dd0c6f87-ae63-46bc-9373-da4088aeba89-kube-api-access-t4r4j\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792696 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-node-log\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-sys-fs\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792764 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-run-k8s-cni-cncf-io\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792900 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a6c1585-8073-46aa-a8d3-9de8a53868a9-iptables-alerter-script\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.792935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-cni-bin\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-hostroot\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-log-socket\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-kubelet-config\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-socket-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793236 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-run-ovn-kubernetes\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5993a850-124f-4910-9e5e-36e75c22589c-multus-daemon-config\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5993a850-124f-4910-9e5e-36e75c22589c-cni-binary-copy\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddl2z\" (UniqueName: \"kubernetes.io/projected/5993a850-124f-4910-9e5e-36e75c22589c-kube-api-access-ddl2z\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.004513 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793296 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-sys-fs\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793429 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-kubelet-config\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-socket-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tttt\" (UniqueName: \"kubernetes.io/projected/798cf565-5257-4b14-8348-47dd6ed08a49-kube-api-access-9tttt\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793600 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-run-ovn-kubernetes\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-var-lib-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793685 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-var-lib-openvswitch\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-cni-netd\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-system-cni-dir\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793801 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-cni-netd\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5993a850-124f-4910-9e5e-36e75c22589c-multus-daemon-config\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-registration-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-system-cni-dir\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/798cf565-5257-4b14-8348-47dd6ed08a49-registration-dir\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.793940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99b9a17e-370f-43bc-8186-fe34fe8a47c6-konnectivity-ca\") pod \"konnectivity-agent-5r44g\" (UID: \"99b9a17e-370f-43bc-8186-fe34fe8a47c6\") " pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:03.005168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-os-release\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-cni-bin\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-systemd\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-kubelet\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794154 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-run-systemd\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cnibin\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-run-netns\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-os-release\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-kubelet\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-system-cni-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794279 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-cnibin\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-host-var-lib-cni-bin\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-etc-kubernetes\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-run-netns\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a6c1585-8073-46aa-a8d3-9de8a53868a9-host-slash\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-system-cni-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovn-node-metrics-cert\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a6c1585-8073-46aa-a8d3-9de8a53868a9-host-slash\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:03.006005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-etc-kubernetes\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-modprobe-d\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysctl-d\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99b9a17e-370f-43bc-8186-fe34fe8a47c6-konnectivity-ca\") pod \"konnectivity-agent-5r44g\" (UID: \"99b9a17e-370f-43bc-8186-fe34fe8a47c6\") " pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-run\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.794484 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-conf-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99b9a17e-370f-43bc-8186-fe34fe8a47c6-agent-certs\") pod \"konnectivity-agent-5r44g\" (UID: \"99b9a17e-370f-43bc-8186-fe34fe8a47c6\") " pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-slash\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5993a850-124f-4910-9e5e-36e75c22589c-multus-conf-dir\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-env-overrides\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.794622 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret podName:97442fa3-b0da-4f2d-a0a7-b5682b9c2402 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:03.294586802 +0000 UTC m=+2.064014708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret") pod "global-pull-secret-syncer-4z46t" (UID: "97442fa3-b0da-4f2d-a0a7-b5682b9c2402") : object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/934d23da-7dc2-40c1-84c1-9784f7a99dde-host-slash\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794640 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xptf7\" (UniqueName: \"kubernetes.io/projected/207a370a-c6ee-42b3-a858-1f9ec07a7191-kube-api-access-xptf7\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:03.006755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-tmp\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovnkube-config\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc62v\" (UniqueName: \"kubernetes.io/projected/934d23da-7dc2-40c1-84c1-9784f7a99dde-kube-api-access-fc62v\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.794862 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.794930 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs podName:207a370a-c6ee-42b3-a858-1f9ec07a7191 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:03.294910955 +0000 UTC m=+2.064338841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs") pod "network-metrics-daemon-hpv4n" (UID: "207a370a-c6ee-42b3-a858-1f9ec07a7191") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.794982 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-env-overrides\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.795384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovnkube-config\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.797584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/934d23da-7dc2-40c1-84c1-9784f7a99dde-ovn-node-metrics-cert\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.797691 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99b9a17e-370f-43bc-8186-fe34fe8a47c6-agent-certs\") pod \"konnectivity-agent-5r44g\" (UID: \"99b9a17e-370f-43bc-8186-fe34fe8a47c6\") " pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.809768 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.809789 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.809802 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cs8gm for pod openshift-network-diagnostics/network-check-target-blsjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:02.809864 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm podName:7e75bd51-6735-4f43-afc5-de7e1d45e98b nodeName:}" failed. No retries permitted until 2026-07-12 16:25:03.309850047 +0000 UTC m=+2.079277945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cs8gm" (UniqueName: "kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm") pod "network-check-target-blsjj" (UID: "7e75bd51-6735-4f43-afc5-de7e1d45e98b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.809877 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8pzf\" (UniqueName: \"kubernetes.io/projected/0a6c1585-8073-46aa-a8d3-9de8a53868a9-kube-api-access-v8pzf\") pod \"iptables-alerter-8js4s\" (UID: \"0a6c1585-8073-46aa-a8d3-9de8a53868a9\") " pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.811969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjj8p\" (UniqueName: \"kubernetes.io/projected/dc38e844-5b6e-4d5c-8a44-08058be6e5ac-kube-api-access-cjj8p\") pod \"multus-additional-cni-plugins-wjtqp\" (UID: \"dc38e844-5b6e-4d5c-8a44-08058be6e5ac\") " pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.812101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4r4j\" (UniqueName: \"kubernetes.io/projected/dd0c6f87-ae63-46bc-9373-da4088aeba89-kube-api-access-t4r4j\") pod \"node-ca-c52dt\" (UID: \"dd0c6f87-ae63-46bc-9373-da4088aeba89\") " pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.812310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tttt\" (UniqueName: \"kubernetes.io/projected/798cf565-5257-4b14-8348-47dd6ed08a49-kube-api-access-9tttt\") pod \"aws-ebs-csi-driver-node-4lmbg\" (UID: \"798cf565-5257-4b14-8348-47dd6ed08a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.007448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.812328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddl2z\" (UniqueName: \"kubernetes.io/projected/5993a850-124f-4910-9e5e-36e75c22589c-kube-api-access-ddl2z\") pod \"multus-wx9bn\" (UID: \"5993a850-124f-4910-9e5e-36e75c22589c\") " pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.813770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xptf7\" (UniqueName: \"kubernetes.io/projected/207a370a-c6ee-42b3-a858-1f9ec07a7191-kube-api-access-xptf7\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.818901 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7lk\" (UniqueName: \"kubernetes.io/projected/cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed-kube-api-access-zz7lk\") pod \"node-resolver-6vlfb\" (UID: \"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed\") " pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.819732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc62v\" (UniqueName: \"kubernetes.io/projected/934d23da-7dc2-40c1-84c1-9784f7a99dde-kube-api-access-fc62v\") pod \"ovnkube-node-zxb5k\" (UID: \"934d23da-7dc2-40c1-84c1-9784f7a99dde\") " pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-modprobe-d\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897536 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysctl-d\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-run\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-tmp\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-lib-modules\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897615 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-run\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-tuned\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-modprobe-d\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysctl-d\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-host\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-kubernetes\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-host\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysctl-conf\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897762 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-lib-modules\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897789 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-kubernetes\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-var-lib-kubelet\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysctl-conf\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-var-lib-kubelet\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysconfig\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897938 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-sysconfig\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-systemd\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-sys\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-systemd\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.897995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smbb\" (UniqueName: \"kubernetes.io/projected/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-kube-api-access-7smbb\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.898028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-sys\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.899784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-etc-tuned\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.899864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-tmp\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.909757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smbb\" (UniqueName: \"kubernetes.io/projected/ba74bcdf-9e07-46a5-ae18-b7a49b0545be-kube-api-access-7smbb\") pod \"tuned-plvmn\" (UID: \"ba74bcdf-9e07-46a5-ae18-b7a49b0545be\") " pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:02.924768 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e86ff064f57baa490d1d3b0d45fbf46.slice/crio-1a69c448bce43278f207d8f5807e9dea0a31610e68605ad1acda03fa837d5ec6 WatchSource:0}: Error finding container 1a69c448bce43278f207d8f5807e9dea0a31610e68605ad1acda03fa837d5ec6: Status 404 returned error can't find the container with id 1a69c448bce43278f207d8f5807e9dea0a31610e68605ad1acda03fa837d5ec6 Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:02.925045 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6203c45afbfc7cc0bd349ce0d548f2.slice/crio-a72754bedc4dbdf2164d12a8f1dcdc8644762c258959422c767259db50bf8a1e WatchSource:0}: Error finding container a72754bedc4dbdf2164d12a8f1dcdc8644762c258959422c767259db50bf8a1e: Status 404 returned error can't find the container with id a72754bedc4dbdf2164d12a8f1dcdc8644762c258959422c767259db50bf8a1e Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.930001 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:02.994388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:03.008740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.000831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8js4s" Jul 12 16:25:03.009688 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.003406 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b9a17e_370f_43bc_8186_fe34fe8a47c6.slice/crio-d94cb26f140c55d8e394a2dd3212a062f6d23b0859e339079b216658faa40628 WatchSource:0}: Error finding container d94cb26f140c55d8e394a2dd3212a062f6d23b0859e339079b216658faa40628: Status 404 returned error can't find the container with id d94cb26f140c55d8e394a2dd3212a062f6d23b0859e339079b216658faa40628 Jul 12 16:25:03.009688 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.008568 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6c1585_8073_46aa_a8d3_9de8a53868a9.slice/crio-fd3f4fe31712ad9eed9a4fec0cd6e509cb1720923d1213fbe638fbd0d2e8db9e WatchSource:0}: Error finding container fd3f4fe31712ad9eed9a4fec0cd6e509cb1720923d1213fbe638fbd0d2e8db9e: Status 404 returned error can't find the container with id fd3f4fe31712ad9eed9a4fec0cd6e509cb1720923d1213fbe638fbd0d2e8db9e Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.024384 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wx9bn" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.031449 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5993a850_124f_4910_9e5e_36e75c22589c.slice/crio-331ee3fdec35ffefb637e7051d27637bde7fef264c26ada8898f6422243f7663 WatchSource:0}: Error finding container 331ee3fdec35ffefb637e7051d27637bde7fef264c26ada8898f6422243f7663: Status 404 returned error can't find the container with id 331ee3fdec35ffefb637e7051d27637bde7fef264c26ada8898f6422243f7663 Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.037385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.044616 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc38e844_5b6e_4d5c_8a44_08058be6e5ac.slice/crio-979677734a05813e409a7880c4584bc0d0990b369a8b995220a4cb36f863bf16 WatchSource:0}: Error finding container 979677734a05813e409a7880c4584bc0d0990b369a8b995220a4cb36f863bf16: Status 404 returned error can't find the container with id 979677734a05813e409a7880c4584bc0d0990b369a8b995220a4cb36f863bf16 Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.056084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.062664 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934d23da_7dc2_40c1_84c1_9784f7a99dde.slice/crio-12d2864fc0c3675d0f86ad4ef963b404dbeedda02124d4380a2ebed3ddbb5536 WatchSource:0}: Error finding container 12d2864fc0c3675d0f86ad4ef963b404dbeedda02124d4380a2ebed3ddbb5536: Status 404 returned error can't find the container with id 12d2864fc0c3675d0f86ad4ef963b404dbeedda02124d4380a2ebed3ddbb5536 Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.069800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6vlfb" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.073533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c52dt" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.076350 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd8c0d4_657b_41e0_a1f7_ed1c9e35d9ed.slice/crio-c80fe054d9d85eb98e8c4487224930177d4e61b948e7795a05f8b81f03a2e338 WatchSource:0}: Error finding container c80fe054d9d85eb98e8c4487224930177d4e61b948e7795a05f8b81f03a2e338: Status 404 returned error can't find the container with id c80fe054d9d85eb98e8c4487224930177d4e61b948e7795a05f8b81f03a2e338 Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.082052 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0c6f87_ae63_46bc_9373_da4088aeba89.slice/crio-1a1ef60552683adbc6eda759d301dd177748ace0fc450759542f9f5e9c2ae1c2 WatchSource:0}: Error finding container 1a1ef60552683adbc6eda759d301dd177748ace0fc450759542f9f5e9c2ae1c2: Status 404 returned error can't find the container with id 1a1ef60552683adbc6eda759d301dd177748ace0fc450759542f9f5e9c2ae1c2 Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.091908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.097862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-plvmn" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.101554 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798cf565_5257_4b14_8348_47dd6ed08a49.slice/crio-bf5be707a791616a951242a8b3493a4c9588dfacb6d3aa7879531f5367190899 WatchSource:0}: Error finding container bf5be707a791616a951242a8b3493a4c9588dfacb6d3aa7879531f5367190899: Status 404 returned error can't find the container with id bf5be707a791616a951242a8b3493a4c9588dfacb6d3aa7879531f5367190899 Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:03.104821 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba74bcdf_9e07_46a5_ae18_b7a49b0545be.slice/crio-afe5443dc6e71554993973e054828837ec8b3db7d083db57db779b9d84657371 WatchSource:0}: Error finding container afe5443dc6e71554993973e054828837ec8b3db7d083db57db779b9d84657371: Status 404 returned error can't find the container with id afe5443dc6e71554993973e054828837ec8b3db7d083db57db779b9d84657371 Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.106243 2575 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-plvmn_openshift-cluster-node-tuning-operator(ba74bcdf-9e07-46a5-ae18-b7a49b0545be): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.106272 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-plvmn" podUID="ba74bcdf-9e07-46a5-ae18-b7a49b0545be" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.300799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.300832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.300949 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:03.305872 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.300950 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:03.307640 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.301017 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs podName:207a370a-c6ee-42b3-a858-1f9ec07a7191 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:04.30099045 +0000 UTC m=+3.070418323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs") pod "network-metrics-daemon-hpv4n" (UID: "207a370a-c6ee-42b3-a858-1f9ec07a7191") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:03.307640 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.301035 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret podName:97442fa3-b0da-4f2d-a0a7-b5682b9c2402 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:04.30102554 +0000 UTC m=+3.070453413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret") pod "global-pull-secret-syncer-4z46t" (UID: "97442fa3-b0da-4f2d-a0a7-b5682b9c2402") : object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:03.432555 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.401436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:03.432555 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.401575 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 16:25:03.432555 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.401593 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 16:25:03.432555 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.401605 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cs8gm for pod openshift-network-diagnostics/network-check-target-blsjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:03.432555 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.401656 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm podName:7e75bd51-6735-4f43-afc5-de7e1d45e98b nodeName:}" failed. No retries permitted until 2026-07-12 16:25:04.401638982 +0000 UTC m=+3.171066857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cs8gm" (UniqueName: "kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm") pod "network-check-target-blsjj" (UID: "7e75bd51-6735-4f43-afc5-de7e1d45e98b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:03.720013 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.719481 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 16:25:03.791904 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.758910 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 16:20:02 +0000 UTC" deadline="2028-03-08 18:16:49.160194259 +0000 UTC" Jul 12 16:25:03.791904 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.758945 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14521h51m45.40125412s" Jul 12 16:25:03.959774 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.909061 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" event={"ID":"798cf565-5257-4b14-8348-47dd6ed08a49","Type":"ContainerStarted","Data":"bf5be707a791616a951242a8b3493a4c9588dfacb6d3aa7879531f5367190899"} Jul 12 16:25:03.959774 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.911948 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c52dt" event={"ID":"dd0c6f87-ae63-46bc-9373-da4088aeba89","Type":"ContainerStarted","Data":"1a1ef60552683adbc6eda759d301dd177748ace0fc450759542f9f5e9c2ae1c2"} Jul 12 16:25:03.959774 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.915249 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wx9bn" event={"ID":"5993a850-124f-4910-9e5e-36e75c22589c","Type":"ContainerStarted","Data":"331ee3fdec35ffefb637e7051d27637bde7fef264c26ada8898f6422243f7663"} Jul 12 16:25:03.959774 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.918883 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8js4s" event={"ID":"0a6c1585-8073-46aa-a8d3-9de8a53868a9","Type":"ContainerStarted","Data":"fd3f4fe31712ad9eed9a4fec0cd6e509cb1720923d1213fbe638fbd0d2e8db9e"} Jul 12 16:25:03.959774 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.932597 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5r44g" event={"ID":"99b9a17e-370f-43bc-8186-fe34fe8a47c6","Type":"ContainerStarted","Data":"d94cb26f140c55d8e394a2dd3212a062f6d23b0859e339079b216658faa40628"} Jul 12 16:25:03.959774 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.952225 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" event={"ID":"0e86ff064f57baa490d1d3b0d45fbf46","Type":"ContainerStarted","Data":"1a69c448bce43278f207d8f5807e9dea0a31610e68605ad1acda03fa837d5ec6"} Jul 12 16:25:03.961716 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.960167 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-plvmn" event={"ID":"ba74bcdf-9e07-46a5-ae18-b7a49b0545be","Type":"ContainerStarted","Data":"afe5443dc6e71554993973e054828837ec8b3db7d083db57db779b9d84657371"} Jul 12 16:25:03.963414 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:03.962854 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-plvmn" podUID="ba74bcdf-9e07-46a5-ae18-b7a49b0545be" Jul 12 16:25:03.979688 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.979659 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6vlfb" event={"ID":"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed","Type":"ContainerStarted","Data":"c80fe054d9d85eb98e8c4487224930177d4e61b948e7795a05f8b81f03a2e338"} Jul 12 16:25:03.998081 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:03.998048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"12d2864fc0c3675d0f86ad4ef963b404dbeedda02124d4380a2ebed3ddbb5536"} Jul 12 16:25:04.003467 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.003433 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" event={"ID":"dc38e844-5b6e-4d5c-8a44-08058be6e5ac","Type":"ContainerStarted","Data":"979677734a05813e409a7880c4584bc0d0990b369a8b995220a4cb36f863bf16"} Jul 12 16:25:04.013345 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.013320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" event={"ID":"4b6203c45afbfc7cc0bd349ce0d548f2","Type":"ContainerStarted","Data":"a72754bedc4dbdf2164d12a8f1dcdc8644762c258959422c767259db50bf8a1e"} Jul 12 16:25:04.071534 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.071322 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 16:25:04.148165 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.144906 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 16:25:04.311009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.310968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:04.311229 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.311029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:04.311229 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.311217 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:04.311367 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.311267 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs podName:207a370a-c6ee-42b3-a858-1f9ec07a7191 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:06.311253207 +0000 UTC m=+5.080681083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs") pod "network-metrics-daemon-hpv4n" (UID: "207a370a-c6ee-42b3-a858-1f9ec07a7191") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:04.311685 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.311668 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:04.311760 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.311724 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret podName:97442fa3-b0da-4f2d-a0a7-b5682b9c2402 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:06.311707007 +0000 UTC m=+5.081134885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret") pod "global-pull-secret-syncer-4z46t" (UID: "97442fa3-b0da-4f2d-a0a7-b5682b9c2402") : object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:04.412004 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.411920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:04.412223 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.412160 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 16:25:04.412223 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.412183 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 16:25:04.412223 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.412211 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cs8gm for pod openshift-network-diagnostics/network-check-target-blsjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:04.412402 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.412268 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm podName:7e75bd51-6735-4f43-afc5-de7e1d45e98b nodeName:}" failed. No retries permitted until 2026-07-12 16:25:06.41224918 +0000 UTC m=+5.181677055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cs8gm" (UniqueName: "kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm") pod "network-check-target-blsjj" (UID: "7e75bd51-6735-4f43-afc5-de7e1d45e98b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:04.759282 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.759219 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 16:20:02 +0000 UTC" deadline="2028-03-16 21:08:27.71757448 +0000 UTC" Jul 12 16:25:04.759282 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.759279 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14716h43m22.958299317s" Jul 12 16:25:04.869097 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.869065 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:04.869560 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.869222 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:04.869670 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.869651 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:04.869776 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.869754 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:04.869834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:04.869820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:04.869908 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:04.869892 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:06.329205 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:06.329097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:06.329205 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:06.329157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:06.329693 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.329241 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:06.329693 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.329326 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret podName:97442fa3-b0da-4f2d-a0a7-b5682b9c2402 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:10.329305357 +0000 UTC m=+9.098733234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret") pod "global-pull-secret-syncer-4z46t" (UID: "97442fa3-b0da-4f2d-a0a7-b5682b9c2402") : object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:06.329693 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.329328 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:06.329693 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.329392 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs podName:207a370a-c6ee-42b3-a858-1f9ec07a7191 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:10.329374508 +0000 UTC m=+9.098802395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs") pod "network-metrics-daemon-hpv4n" (UID: "207a370a-c6ee-42b3-a858-1f9ec07a7191") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:06.430178 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:06.430137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:06.430356 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.430335 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 16:25:06.430415 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.430357 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 16:25:06.430415 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.430372 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cs8gm for pod openshift-network-diagnostics/network-check-target-blsjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:06.430535 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.430433 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm podName:7e75bd51-6735-4f43-afc5-de7e1d45e98b nodeName:}" failed. No retries permitted until 2026-07-12 16:25:10.430415084 +0000 UTC m=+9.199842960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cs8gm" (UniqueName: "kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm") pod "network-check-target-blsjj" (UID: "7e75bd51-6735-4f43-afc5-de7e1d45e98b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:06.869659 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:06.869619 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:06.869659 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:06.869655 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:06.869902 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.869778 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:06.870230 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.870204 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:06.870323 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:06.870259 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:06.870376 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:06.870350 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:08.869530 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:08.869035 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:08.869530 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:08.869144 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:08.869530 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:08.869173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:08.869530 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:08.869393 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:08.869530 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:08.869039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:08.869530 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:08.869488 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:10.364556 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:10.363879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:10.364556 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:10.363937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:10.364556 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.364093 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:10.364556 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.364155 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs podName:207a370a-c6ee-42b3-a858-1f9ec07a7191 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:18.364136484 +0000 UTC m=+17.133564362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs") pod "network-metrics-daemon-hpv4n" (UID: "207a370a-c6ee-42b3-a858-1f9ec07a7191") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:10.364556 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.364093 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:10.364556 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.364531 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret podName:97442fa3-b0da-4f2d-a0a7-b5682b9c2402 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:18.364513948 +0000 UTC m=+17.133941828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret") pod "global-pull-secret-syncer-4z46t" (UID: "97442fa3-b0da-4f2d-a0a7-b5682b9c2402") : object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:10.465029 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:10.464992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:10.465198 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.465155 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 16:25:10.465198 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.465172 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 16:25:10.465198 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.465181 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cs8gm for pod openshift-network-diagnostics/network-check-target-blsjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:10.465318 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.465279 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm podName:7e75bd51-6735-4f43-afc5-de7e1d45e98b nodeName:}" failed. No retries permitted until 2026-07-12 16:25:18.465259259 +0000 UTC m=+17.234687135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cs8gm" (UniqueName: "kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm") pod "network-check-target-blsjj" (UID: "7e75bd51-6735-4f43-afc5-de7e1d45e98b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:10.869154 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:10.868616 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:10.869154 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:10.868801 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:10.869154 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.868837 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:10.869154 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.868904 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:10.869154 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:10.868956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:10.869154 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:10.869037 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:12.869072 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:12.869032 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:12.869554 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:12.869032 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:12.869554 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:12.869162 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:12.869554 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:12.869032 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:12.869554 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:12.869323 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:12.869554 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:12.869260 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:14.869486 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:14.869446 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:14.869901 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:14.869462 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:14.869901 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:14.869470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:14.869901 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:14.869609 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:14.869901 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:14.869717 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:14.869901 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:14.869794 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:16.869260 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:16.869227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:16.869741 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:16.869282 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:16.869741 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:16.869390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:16.869741 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:16.869401 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:16.869741 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:16.869468 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:16.869741 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:16.869486 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:18.425006 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:18.424966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:18.425006 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:18.425024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:18.425609 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.425143 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:18.425609 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.425176 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:18.425609 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.425235 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret podName:97442fa3-b0da-4f2d-a0a7-b5682b9c2402 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:34.425215641 +0000 UTC m=+33.194643519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret") pod "global-pull-secret-syncer-4z46t" (UID: "97442fa3-b0da-4f2d-a0a7-b5682b9c2402") : object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:18.425609 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.425257 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs podName:207a370a-c6ee-42b3-a858-1f9ec07a7191 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:34.425245383 +0000 UTC m=+33.194673257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs") pod "network-metrics-daemon-hpv4n" (UID: "207a370a-c6ee-42b3-a858-1f9ec07a7191") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:18.525822 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:18.525785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:18.525993 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.525966 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 16:25:18.525993 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.525989 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 16:25:18.526103 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.526001 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cs8gm for pod openshift-network-diagnostics/network-check-target-blsjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:18.526103 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.526064 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm podName:7e75bd51-6735-4f43-afc5-de7e1d45e98b nodeName:}" failed. No retries permitted until 2026-07-12 16:25:34.526044914 +0000 UTC m=+33.295472792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cs8gm" (UniqueName: "kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm") pod "network-check-target-blsjj" (UID: "7e75bd51-6735-4f43-afc5-de7e1d45e98b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:18.868873 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:18.868831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:18.869018 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:18.868843 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:18.869018 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.868976 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:18.869128 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:18.868852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:18.869128 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.869037 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:18.869219 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:18.869136 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:20.870262 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:20.869714 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:20.870262 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:20.870072 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:20.870262 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:20.870148 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:20.871172 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:20.871084 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:20.871172 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:20.871154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:20.871329 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:20.871282 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:21.049875 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.049669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" event={"ID":"798cf565-5257-4b14-8348-47dd6ed08a49","Type":"ContainerStarted","Data":"103d5a41b6eb7c8e70f9be42d296239fcdc0f1b92e3b1fa244a12a37502bd58a"} Jul 12 16:25:21.051467 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.051417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c52dt" event={"ID":"dd0c6f87-ae63-46bc-9373-da4088aeba89","Type":"ContainerStarted","Data":"6708f5ea6b798689c019fe633dba20b3c10972e01ac169d35501abbf2460f2e7"} Jul 12 16:25:21.053304 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.053279 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wx9bn" event={"ID":"5993a850-124f-4910-9e5e-36e75c22589c","Type":"ContainerStarted","Data":"0611c248ab72f45f6fda292b4b2ff85016f667294da4f2c3918143cde9cbf445"} Jul 12 16:25:21.055292 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.055231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5r44g" event={"ID":"99b9a17e-370f-43bc-8186-fe34fe8a47c6","Type":"ContainerStarted","Data":"ef9bb1502119fca3e3da1b5574bf23eca14980d270a0c9f3c0fc89ecb08fbafa"} Jul 12 16:25:21.057184 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.057162 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e86ff064f57baa490d1d3b0d45fbf46" containerID="1789d8210e14629fe3ea46a74cfbded6531ddc029e5045ae720912359ec4884e" exitCode=0 Jul 12 16:25:21.057184 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.057255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" event={"ID":"0e86ff064f57baa490d1d3b0d45fbf46","Type":"ContainerDied","Data":"1789d8210e14629fe3ea46a74cfbded6531ddc029e5045ae720912359ec4884e"} Jul 12 16:25:21.058669 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.058629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6vlfb" event={"ID":"cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed","Type":"ContainerStarted","Data":"b7f2f023b93c62edf3f03d98b69d331d1d217d2d3a01ee1039e0371472bf4fe1"} Jul 12 16:25:21.061566 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.061509 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:25:21.066574 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.062219 2575 generic.go:356] "Generic (PLEG): container finished" podID="934d23da-7dc2-40c1-84c1-9784f7a99dde" containerID="d3ecd209fe1f9fdc3bc44222cb1baa78e3365fbb91832cceee44465cb499f9d9" exitCode=1 Jul 12 16:25:21.066574 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.062322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"c7cdad448bfbcccf44bb2955267a9304d9934e02dee8d45e9c6a25088e280067"} Jul 12 16:25:21.066915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.066758 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"4fa6cfcb5427edf5f06d72ee01612f15a76671a63f147bd21dba26b378a6e868"} Jul 12 16:25:21.066915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.066835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"53ec95be74a7ce1ed5c1e772c06ecbeaa744286b20b51a4cfc754dbab919859b"} Jul 12 16:25:21.067004 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.064160 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc38e844-5b6e-4d5c-8a44-08058be6e5ac" containerID="f0b8634adca1208cdb14cdf5de7d82ae20468db9faeb23bcc25b9bcc33eddde5" exitCode=0 Jul 12 16:25:21.067056 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.067031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerDied","Data":"d3ecd209fe1f9fdc3bc44222cb1baa78e3365fbb91832cceee44465cb499f9d9"} Jul 12 16:25:21.067155 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.067141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"db3ddba2c92478985694bbc8f2c9c48a0e7f67db51dd4c05bbbbd63d0163d475"} Jul 12 16:25:21.067256 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.067235 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" event={"ID":"dc38e844-5b6e-4d5c-8a44-08058be6e5ac","Type":"ContainerDied","Data":"f0b8634adca1208cdb14cdf5de7d82ae20468db9faeb23bcc25b9bcc33eddde5"} Jul 12 16:25:21.069306 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.069285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" event={"ID":"4b6203c45afbfc7cc0bd349ce0d548f2","Type":"ContainerStarted","Data":"9e3260cc6e96a54dced82ccc047fd675cbe2639eac20b725b838e11049cf6a9f"} Jul 12 16:25:21.071442 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.070912 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c52dt" podStartSLOduration=1.830209819 podStartE2EDuration="19.070895635s" totalImagesPullingTime="17.240685816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:21.070325337 +0000 UTC m=+19.839753233" watchObservedRunningTime="2026-07-12 16:25:21.070895635 +0000 UTC m=+19.840323509" Jul 12 16:25:21.120466 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.120343 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wx9bn" podStartSLOduration=2.3860660879999998 podStartE2EDuration="20.120324178s" totalImagesPullingTime="17.73425809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:21.102900495 +0000 UTC m=+19.872328396" watchObservedRunningTime="2026-07-12 16:25:21.120324178 +0000 UTC m=+19.889752099" Jul 12 16:25:21.120773 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.120730 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6vlfb" podStartSLOduration=1.873564049 podStartE2EDuration="19.12071883s" totalImagesPullingTime="17.247154781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:21.120280561 +0000 UTC m=+19.889708480" watchObservedRunningTime="2026-07-12 16:25:21.12071883 +0000 UTC m=+19.890146725" Jul 12 16:25:21.161947 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.161879 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-123.ec2.internal" podStartSLOduration=19.161858258 podStartE2EDuration="19.161858258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:21.138330649 +0000 UTC m=+19.907758548" watchObservedRunningTime="2026-07-12 16:25:21.161858258 +0000 UTC m=+19.931286154" Jul 12 16:25:21.183654 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.183600 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5r44g" podStartSLOduration=2.865245144 podStartE2EDuration="20.183579041s" totalImagesPullingTime="17.318333897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:21.183219428 +0000 UTC m=+19.952647324" watchObservedRunningTime="2026-07-12 16:25:21.183579041 +0000 UTC m=+19.953006946" Jul 12 16:25:21.226378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.226344 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:21.393012 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.392983 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:21.459532 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.459330 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:21.776657 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.776623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:21.876763 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.876727 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:21.942039 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.942012 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:21.942620 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:21.942597 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:22.005896 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.005864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:22.073858 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.073825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8js4s" event={"ID":"0a6c1585-8073-46aa-a8d3-9de8a53868a9","Type":"ContainerStarted","Data":"5ea68f755b60ea2874f7439a8aa4bd0fb51504e551b61898dcbf11c01e018ebf"} Jul 12 16:25:22.077461 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.077438 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:25:22.077836 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.077813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"b95ff0df67f0fc6becf283319a4f180fe69c3947c008e48c7ed7b2301018c7b1"} Jul 12 16:25:22.079465 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.079362 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:22.079465 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.079437 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5r44g" Jul 12 16:25:22.089728 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.089682 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8js4s" podStartSLOduration=3.775843083 podStartE2EDuration="21.089666077s" totalImagesPullingTime="17.313822994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:22.089599538 +0000 UTC m=+20.859027435" watchObservedRunningTime="2026-07-12 16:25:22.089666077 +0000 UTC m=+20.859093975" Jul 12 16:25:22.119726 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.119701 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 12 16:25:22.378026 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.377914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:22.477122 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.477063 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:22.577368 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.577299 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:22.678160 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.678122 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:22.764986 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.764860 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T16:25:22.119722985Z","UUID":"9067f522-dced-4398-9d9a-9e0b427ad571","Handler":null,"Name":"","Endpoint":""} Jul 12 16:25:22.766601 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.766580 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 12 16:25:22.766740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.766621 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 12 16:25:22.777376 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.777347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:22.869952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.869387 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:22.869952 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:22.869509 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:22.869952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.869889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:22.870285 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:22.869989 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:22.870285 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:22.870031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:22.870285 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:22.870139 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:23.082019 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:23.081944 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" event={"ID":"798cf565-5257-4b14-8348-47dd6ed08a49","Type":"ContainerStarted","Data":"eed81ebdb0e10cd7c8829c85d6cc5cdbaf5f399fe3cb3187a8d5e5f4884c0263"} Jul 12 16:25:23.084013 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:23.083983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" event={"ID":"0e86ff064f57baa490d1d3b0d45fbf46","Type":"ContainerStarted","Data":"547349da93b23529b0316508c020210b3538848aae533d10588c813c4ee904f3"} Jul 12 16:25:23.099592 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:23.099554 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-123.ec2.internal" podStartSLOduration=21.099541141 podStartE2EDuration="21.099541141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:23.099515689 +0000 UTC m=+21.868943585" watchObservedRunningTime="2026-07-12 16:25:23.099541141 +0000 UTC m=+21.868969036" Jul 12 16:25:23.172298 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:23.172262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:23.975972 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:23.975938 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:24.168953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:24.168925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:24.274039 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:24.273968 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:24.569869 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:24.569781 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:24.769532 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:24.769493 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:24.868714 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:24.868638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:24.868714 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:24.868670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:24.868714 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:24.868670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:24.868964 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:24.868774 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:24.868964 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:24.868832 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:24.868964 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:24.868883 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:25.174484 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:25.174414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:25.579583 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:25.579546 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:25.770357 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:25.770313 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:26.869433 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:26.869400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:26.869433 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:26.869423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:26.869941 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:26.869400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:26.869941 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:26.869524 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:26.869941 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:26.869586 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:26.869941 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:26.869651 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:28.097057 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.096795 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" event={"ID":"798cf565-5257-4b14-8348-47dd6ed08a49","Type":"ContainerStarted","Data":"9811e80dc021fd4c2ae3da63a4bd3279874bcf140d64c796641cb242f52a9388"} Jul 12 16:25:28.098310 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.098286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-plvmn" event={"ID":"ba74bcdf-9e07-46a5-ae18-b7a49b0545be","Type":"ContainerStarted","Data":"842aafcfa6e496f81e3ef60ad7a80d669d8e2eed587cc6f889ce90759b19cbed"} Jul 12 16:25:28.101200 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.101164 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:25:28.101559 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.101540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"75e093b5eaced25b66f01713d2d46d71c74860d0439d44912c7256d17e2c5e4d"} Jul 12 16:25:28.103114 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.103092 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc38e844-5b6e-4d5c-8a44-08058be6e5ac" containerID="849ecad5e54270a9cec3a060ad07f8f81200a14cc2918b170a0d57dbc6bcf1cb" exitCode=0 Jul 12 16:25:28.103229 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.103124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" event={"ID":"dc38e844-5b6e-4d5c-8a44-08058be6e5ac","Type":"ContainerDied","Data":"849ecad5e54270a9cec3a060ad07f8f81200a14cc2918b170a0d57dbc6bcf1cb"} Jul 12 16:25:28.134769 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.134727 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lmbg" podStartSLOduration=2.785280206 podStartE2EDuration="26.134713912s" totalImagesPullingTime="23.349433706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:28.116317588 +0000 UTC m=+26.885745494" watchObservedRunningTime="2026-07-12 16:25:28.134713912 +0000 UTC m=+26.904141807" Jul 12 16:25:28.151228 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.151170 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-plvmn" podStartSLOduration=2.045686195 podStartE2EDuration="26.151158205s" totalImagesPullingTime="24.10547201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-12 16:25:28.150794388 +0000 UTC m=+26.920222283" watchObservedRunningTime="2026-07-12 16:25:28.151158205 +0000 UTC m=+26.920586100" Jul 12 16:25:28.869023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.868977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:28.869023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.868998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:28.869023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:28.868999 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:28.869312 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:28.869106 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:28.869312 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:28.869202 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:28.869312 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:28.869273 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:30.110331 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.110117 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:25:30.111230 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.110667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"5c4f1708785d0248103ab0873a2d1080eabeb1657d6302b13e1cc2e5b22126a2"} Jul 12 16:25:30.111230 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.110895 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:30.111230 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.111094 2575 scope.go:117] "RemoveContainer" containerID="d3ecd209fe1f9fdc3bc44222cb1baa78e3365fbb91832cceee44465cb499f9d9" Jul 12 16:25:30.112935 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.112916 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc38e844-5b6e-4d5c-8a44-08058be6e5ac" containerID="8c34022620ef61b380669357c51e749a5d18ed929e1d9384c113a8d6f8145d94" exitCode=0 Jul 12 16:25:30.113042 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.112949 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" event={"ID":"dc38e844-5b6e-4d5c-8a44-08058be6e5ac","Type":"ContainerDied","Data":"8c34022620ef61b380669357c51e749a5d18ed929e1d9384c113a8d6f8145d94"} Jul 12 16:25:30.127468 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.127448 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:30.869025 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.868998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:30.869025 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.869017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:30.869175 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:30.869020 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:30.869175 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:30.869102 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:30.869175 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:30.869160 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:30.869282 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:30.869266 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:31.118860 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.118781 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:25:31.119264 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.119169 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" event={"ID":"934d23da-7dc2-40c1-84c1-9784f7a99dde","Type":"ContainerStarted","Data":"ec4de85b76f7f6279ab288707c0c1b579276da04abd67aa5457cac2da33a3b69"} Jul 12 16:25:31.119264 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.119250 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 12 16:25:31.119523 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.119503 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:31.121616 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.121592 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc38e844-5b6e-4d5c-8a44-08058be6e5ac" containerID="a353d0889d4d02e7d549aec0c34fe4a187431bf0b2767e65ffda2ddfcff3e3ae" exitCode=0 Jul 12 16:25:31.121701 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.121624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" event={"ID":"dc38e844-5b6e-4d5c-8a44-08058be6e5ac","Type":"ContainerDied","Data":"a353d0889d4d02e7d549aec0c34fe4a187431bf0b2767e65ffda2ddfcff3e3ae"} Jul 12 16:25:31.135409 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.135388 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:31.145613 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.145573 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" podStartSLOduration=11.799749523 podStartE2EDuration="29.145560213s" totalImagesPullingTime="17.34581069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:31.144440679 +0000 UTC m=+29.913868575" watchObservedRunningTime="2026-07-12 16:25:31.145560213 +0000 UTC m=+29.914988107" Jul 12 16:25:31.511891 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.511855 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:25:31.783496 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.783379 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:31.903975 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:31.903941 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:32.002458 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.002423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:32.125703 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.125622 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:32.218329 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.218302 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:32.316245 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.316158 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:32.714876 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.714625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-blsjj"] Jul 12 16:25:32.715043 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.714998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:32.715132 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:32.715109 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:32.717404 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.717379 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4z46t"] Jul 12 16:25:32.717654 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.717636 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:32.717745 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:32.717721 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:32.718167 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.718144 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hpv4n"] Jul 12 16:25:32.718284 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:32.718258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:32.718386 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:32.718362 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:33.869490 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:33.869452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:33.869490 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:33.869488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:33.869873 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:33.869452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:33.869873 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:33.869579 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:33.869873 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:33.869632 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:33.869873 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:33.869723 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:34.442386 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:34.442349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:34.442386 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:34.442393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:34.442627 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:34.442504 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:34.442627 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:34.442569 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs podName:207a370a-c6ee-42b3-a858-1f9ec07a7191 nodeName:}" failed. No retries permitted until 2026-07-12 16:26:06.442550224 +0000 UTC m=+65.211978119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs") pod "network-metrics-daemon-hpv4n" (UID: "207a370a-c6ee-42b3-a858-1f9ec07a7191") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 16:25:34.442627 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:34.442504 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:34.442760 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:34.442661 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret podName:97442fa3-b0da-4f2d-a0a7-b5682b9c2402 nodeName:}" failed. No retries permitted until 2026-07-12 16:26:06.442640265 +0000 UTC m=+65.212068142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret") pod "global-pull-secret-syncer-4z46t" (UID: "97442fa3-b0da-4f2d-a0a7-b5682b9c2402") : object "kube-system"/"original-pull-secret" not registered Jul 12 16:25:34.542817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:34.542784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:34.543003 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:34.542955 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 16:25:34.543003 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:34.542973 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 16:25:34.543003 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:34.542982 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cs8gm for pod openshift-network-diagnostics/network-check-target-blsjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:34.543160 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:34.543030 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm podName:7e75bd51-6735-4f43-afc5-de7e1d45e98b nodeName:}" failed. No retries permitted until 2026-07-12 16:26:06.543010721 +0000 UTC m=+65.312438608 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cs8gm" (UniqueName: "kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm") pod "network-check-target-blsjj" (UID: "7e75bd51-6735-4f43-afc5-de7e1d45e98b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 16:25:34.544118 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:34.544085 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:34.646293 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:34.646251 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:35.857138 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:35.857116 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:35.869385 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:35.869282 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:35.869385 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:35.869283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:35.869555 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:35.869406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:35.869555 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:35.869408 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:35.869555 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:35.869502 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:35.869757 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:35.869599 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:35.953515 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:35.953482 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:36.135382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:36.135287 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc38e844-5b6e-4d5c-8a44-08058be6e5ac" containerID="33aa766a928aa21a1c73e7f2da493579baa18fc6deeba49bc33d039f28721dbb" exitCode=0 Jul 12 16:25:36.135382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:36.135340 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" event={"ID":"dc38e844-5b6e-4d5c-8a44-08058be6e5ac","Type":"ContainerDied","Data":"33aa766a928aa21a1c73e7f2da493579baa18fc6deeba49bc33d039f28721dbb"} Jul 12 16:25:37.140080 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:37.140049 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc38e844-5b6e-4d5c-8a44-08058be6e5ac" containerID="11395b1ad07e1a70fd028e33885669bfcf22e6a3426c2a93c0eca5bc0bdb1b0a" exitCode=0 Jul 12 16:25:37.140618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:37.140089 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" event={"ID":"dc38e844-5b6e-4d5c-8a44-08058be6e5ac","Type":"ContainerDied","Data":"11395b1ad07e1a70fd028e33885669bfcf22e6a3426c2a93c0eca5bc0bdb1b0a"} Jul 12 16:25:37.869127 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:37.869096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:37.869295 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:37.869096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:37.869295 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:37.869234 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:37.869295 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:37.869280 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:37.869418 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:37.869107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:37.869418 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:37.869406 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:38.146788 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:38.146711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" event={"ID":"dc38e844-5b6e-4d5c-8a44-08058be6e5ac","Type":"ContainerStarted","Data":"3052b4f81f7137e993f206259280c4a894176346acf040cd05c573f16c87bff3"} Jul 12 16:25:38.247146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:38.247095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:38.392817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:38.392783 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:38.460569 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:38.460531 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:39.869381 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:39.869343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:39.869846 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:39.869347 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:39.869846 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:39.869435 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blsjj" podUID="7e75bd51-6735-4f43-afc5-de7e1d45e98b" Jul 12 16:25:39.869846 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:39.869492 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z46t" podUID="97442fa3-b0da-4f2d-a0a7-b5682b9c2402" Jul 12 16:25:39.869846 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:39.869542 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:39.869846 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:39.869603 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpv4n" podUID="207a370a-c6ee-42b3-a858-1f9ec07a7191" Jul 12 16:25:41.233492 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.233458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:41.286769 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.286577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:41.527326 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.527259 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-123.ec2.internal" event="NodeReady" Jul 12 16:25:41.527454 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.527363 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 12 16:25:41.568387 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.568339 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wjtqp" podStartSLOduration=10.638016715 podStartE2EDuration="40.568321505s" totalImagesPullingTime="29.93030479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:38.197952105 +0000 UTC m=+36.967380001" watchObservedRunningTime="2026-07-12 16:25:41.568321505 +0000 UTC m=+40.337749401" Jul 12 16:25:41.568953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.568938 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6snlj"] Jul 12 16:25:41.605957 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.605932 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-th26f"] Jul 12 16:25:41.606100 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.606083 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.609953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.609885 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-trdc7\"" Jul 12 16:25:41.610084 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.609977 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 12 16:25:41.610834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.610815 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 12 16:25:41.632497 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.632476 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6snlj"] Jul 12 16:25:41.632621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.632509 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-th26f"] Jul 12 16:25:41.632621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.632531 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8hnw8"] Jul 12 16:25:41.632621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.632606 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.636283 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.636262 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p9km6\"" Jul 12 16:25:41.636404 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.636336 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 12 16:25:41.636404 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.636344 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 12 16:25:41.636404 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.636365 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 12 16:25:41.636404 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.636367 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 12 16:25:41.654937 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.654919 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8hnw8"] Jul 12 16:25:41.655026 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.655005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8hnw8" Jul 12 16:25:41.657375 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.657357 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 12 16:25:41.657539 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.657527 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b5bf5\"" Jul 12 16:25:41.657699 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.657682 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 12 16:25:41.657740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.657700 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 12 16:25:41.690288 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.690260 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:41.695402 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.695385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0dfc77-915e-424b-959b-ab873bf51dbe-tmp-dir\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.695474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.695422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b0dfc77-915e-424b-959b-ab873bf51dbe-metrics-tls\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.695474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.695439 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6lz\" (UniqueName: \"kubernetes.io/projected/6b0dfc77-915e-424b-959b-ab873bf51dbe-kube-api-access-ll6lz\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.695474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.695455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b0dfc77-915e-424b-959b-ab873bf51dbe-config-volume\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.742802 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.742765 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:41.795968 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.795886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b0dfc77-915e-424b-959b-ab873bf51dbe-config-volume\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.795968 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.795915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-data-volume\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.796091 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.795968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.796091 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796015 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0dfc77-915e-424b-959b-ab873bf51dbe-tmp-dir\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.796091 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.796091 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxn4x\" (UniqueName: \"kubernetes.io/projected/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-kube-api-access-dxn4x\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.796091 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e93a099-983c-4ae5-8d65-b25c4d30884b-cert\") pod \"ingress-canary-8hnw8\" (UID: \"0e93a099-983c-4ae5-8d65-b25c4d30884b\") " pod="openshift-ingress-canary/ingress-canary-8hnw8" Jul 12 16:25:41.796352 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsg8w\" (UniqueName: \"kubernetes.io/projected/0e93a099-983c-4ae5-8d65-b25c4d30884b-kube-api-access-wsg8w\") pod \"ingress-canary-8hnw8\" (UID: \"0e93a099-983c-4ae5-8d65-b25c4d30884b\") " pod="openshift-ingress-canary/ingress-canary-8hnw8" Jul 12 16:25:41.796352 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b0dfc77-915e-424b-959b-ab873bf51dbe-metrics-tls\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.796352 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6lz\" (UniqueName: \"kubernetes.io/projected/6b0dfc77-915e-424b-959b-ab873bf51dbe-kube-api-access-ll6lz\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.796352 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-crio-socket\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.796352 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796296 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0dfc77-915e-424b-959b-ab873bf51dbe-tmp-dir\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.796590 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.796572 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b0dfc77-915e-424b-959b-ab873bf51dbe-config-volume\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.800153 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.800130 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b0dfc77-915e-424b-959b-ab873bf51dbe-metrics-tls\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.806372 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.806351 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6lz\" (UniqueName: \"kubernetes.io/projected/6b0dfc77-915e-424b-959b-ab873bf51dbe-kube-api-access-ll6lz\") pod \"dns-default-6snlj\" (UID: \"6b0dfc77-915e-424b-959b-ab873bf51dbe\") " pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.870147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.870121 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:25:41.870324 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.870308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:25:41.870466 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.870446 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:25:41.872854 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.872832 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 16:25:41.872932 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.872918 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 16:25:41.872987 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.872970 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 16:25:41.873039 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.873024 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kpl6g\"" Jul 12 16:25:41.873113 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.873089 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 16:25:41.873254 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.873240 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xb45\"" Jul 12 16:25:41.897183 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.897275 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxn4x\" (UniqueName: \"kubernetes.io/projected/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-kube-api-access-dxn4x\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.897275 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e93a099-983c-4ae5-8d65-b25c4d30884b-cert\") pod \"ingress-canary-8hnw8\" (UID: \"0e93a099-983c-4ae5-8d65-b25c4d30884b\") " pod="openshift-ingress-canary/ingress-canary-8hnw8" Jul 12 16:25:41.897376 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsg8w\" (UniqueName: \"kubernetes.io/projected/0e93a099-983c-4ae5-8d65-b25c4d30884b-kube-api-access-wsg8w\") pod \"ingress-canary-8hnw8\" (UID: \"0e93a099-983c-4ae5-8d65-b25c4d30884b\") " pod="openshift-ingress-canary/ingress-canary-8hnw8" Jul 12 16:25:41.897376 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-crio-socket\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.897376 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-data-volume\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.897518 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897498 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-crio-socket\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.897556 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.897736 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-data-volume\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.897823 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.897720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.899542 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.899521 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e93a099-983c-4ae5-8d65-b25c4d30884b-cert\") pod \"ingress-canary-8hnw8\" (UID: \"0e93a099-983c-4ae5-8d65-b25c4d30884b\") " pod="openshift-ingress-canary/ingress-canary-8hnw8" Jul 12 16:25:41.899615 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.899539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.905656 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.905633 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxn4x\" (UniqueName: \"kubernetes.io/projected/774f43ee-8efb-4b23-9ffc-5d6ee08f1d18-kube-api-access-dxn4x\") pod \"insights-runtime-extractor-th26f\" (UID: \"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18\") " pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.906054 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.906037 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsg8w\" (UniqueName: \"kubernetes.io/projected/0e93a099-983c-4ae5-8d65-b25c4d30884b-kube-api-access-wsg8w\") pod \"ingress-canary-8hnw8\" (UID: \"0e93a099-983c-4ae5-8d65-b25c4d30884b\") " pod="openshift-ingress-canary/ingress-canary-8hnw8" Jul 12 16:25:41.916798 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.916776 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:41.941210 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.941181 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-th26f" Jul 12 16:25:41.975932 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:41.975906 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8hnw8" Jul 12 16:25:42.122183 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.121762 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6snlj"] Jul 12 16:25:42.122183 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.121874 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-th26f"] Jul 12 16:25:42.126234 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:42.126205 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0dfc77_915e_424b_959b_ab873bf51dbe.slice/crio-b18a7935235b71e07c6a22d4a889f13b8cb1a023bd98a32e8abe98b305530daa WatchSource:0}: Error finding container b18a7935235b71e07c6a22d4a889f13b8cb1a023bd98a32e8abe98b305530daa: Status 404 returned error can't find the container with id b18a7935235b71e07c6a22d4a889f13b8cb1a023bd98a32e8abe98b305530daa Jul 12 16:25:42.126463 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:42.126444 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774f43ee_8efb_4b23_9ffc_5d6ee08f1d18.slice/crio-c679e2c4bc01e4c2f5f3e414b62ae511092b47c241f3e4e8375bffd52e429326 WatchSource:0}: Error finding container c679e2c4bc01e4c2f5f3e414b62ae511092b47c241f3e4e8375bffd52e429326: Status 404 returned error can't find the container with id c679e2c4bc01e4c2f5f3e414b62ae511092b47c241f3e4e8375bffd52e429326 Jul 12 16:25:42.135552 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.135527 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8hnw8"] Jul 12 16:25:42.139122 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:42.139101 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e93a099_983c_4ae5_8d65_b25c4d30884b.slice/crio-a2b54e424e64acfc4bb9ee34abdca02790f56351bd980f2a4799e6cd7c4bb619 WatchSource:0}: Error finding container a2b54e424e64acfc4bb9ee34abdca02790f56351bd980f2a4799e6cd7c4bb619: Status 404 returned error can't find the container with id a2b54e424e64acfc4bb9ee34abdca02790f56351bd980f2a4799e6cd7c4bb619 Jul 12 16:25:42.156067 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.156038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8hnw8" event={"ID":"0e93a099-983c-4ae5-8d65-b25c4d30884b","Type":"ContainerStarted","Data":"a2b54e424e64acfc4bb9ee34abdca02790f56351bd980f2a4799e6cd7c4bb619"} Jul 12 16:25:42.157042 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.157020 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-th26f" event={"ID":"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18","Type":"ContainerStarted","Data":"c679e2c4bc01e4c2f5f3e414b62ae511092b47c241f3e4e8375bffd52e429326"} Jul 12 16:25:42.157963 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.157943 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6snlj" event={"ID":"6b0dfc77-915e-424b-959b-ab873bf51dbe","Type":"ContainerStarted","Data":"b18a7935235b71e07c6a22d4a889f13b8cb1a023bd98a32e8abe98b305530daa"} Jul 12 16:25:42.179903 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.179877 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:42.281325 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.281296 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:42.678592 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.678486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:42.732178 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:42.732076 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:43.027202 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:43.027033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:43.080930 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:43.080896 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:43.650985 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:43.650951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:43.704107 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:43.704048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:45.014892 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.014693 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:45.116626 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.116595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:45.180396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.180368 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:45.182638 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.182609 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49"] Jul 12 16:25:45.310404 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.310317 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49"] Jul 12 16:25:45.310549 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.310446 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" Jul 12 16:25:45.313145 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.313069 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 12 16:25:45.313304 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.313146 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lchs8\"" Jul 12 16:25:45.425294 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.425254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c7fccd46-c827-4d70-827d-70d54b527970-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-g5s49\" (UID: \"c7fccd46-c827-4d70-827d-70d54b527970\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" Jul 12 16:25:45.525888 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.525794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c7fccd46-c827-4d70-827d-70d54b527970-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-g5s49\" (UID: \"c7fccd46-c827-4d70-827d-70d54b527970\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" Jul 12 16:25:45.526120 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:45.526001 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 12 16:25:45.526120 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:45.526078 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fccd46-c827-4d70-827d-70d54b527970-tls-certificates podName:c7fccd46-c827-4d70-827d-70d54b527970 nodeName:}" failed. No retries permitted until 2026-07-12 16:25:46.026058117 +0000 UTC m=+44.795485991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c7fccd46-c827-4d70-827d-70d54b527970-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-g5s49" (UID: "c7fccd46-c827-4d70-827d-70d54b527970") : secret "prometheus-operator-admission-webhook-tls" not found Jul 12 16:25:45.664989 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.664907 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:45.953784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:45.953755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:46.030881 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:46.030835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c7fccd46-c827-4d70-827d-70d54b527970-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-g5s49\" (UID: \"c7fccd46-c827-4d70-827d-70d54b527970\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" Jul 12 16:25:46.034646 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:46.034619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c7fccd46-c827-4d70-827d-70d54b527970-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-g5s49\" (UID: \"c7fccd46-c827-4d70-827d-70d54b527970\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" Jul 12 16:25:46.053481 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:46.053453 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:46.221356 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:46.221264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" Jul 12 16:25:46.631872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:46.631779 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:46.736888 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:46.736856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:47.135445 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:47.135401 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:47.237688 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:47.237654 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:48.179779 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:48.179757 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49"] Jul 12 16:25:48.182851 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:48.182824 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fccd46_c827_4d70_827d_70d54b527970.slice/crio-bc5a9e683f0796ff6cfe48de8b9eedf0be4e015043cdd8311fbc14f68cf97cfc WatchSource:0}: Error finding container bc5a9e683f0796ff6cfe48de8b9eedf0be4e015043cdd8311fbc14f68cf97cfc: Status 404 returned error can't find the container with id bc5a9e683f0796ff6cfe48de8b9eedf0be4e015043cdd8311fbc14f68cf97cfc Jul 12 16:25:49.178550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:49.178511 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8hnw8" event={"ID":"0e93a099-983c-4ae5-8d65-b25c4d30884b","Type":"ContainerStarted","Data":"fc1128aaa9d97efe18f248f93ffe16c29da230c295a4a231d7aca51beb2f232e"} Jul 12 16:25:49.180663 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:49.180633 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6snlj" event={"ID":"6b0dfc77-915e-424b-959b-ab873bf51dbe","Type":"ContainerStarted","Data":"2a1e43af1c83e21175dbef5f08f9247e28ffd3447039bf08bc21c8f0e13d76d5"} Jul 12 16:25:49.181036 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:49.180688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6snlj" Jul 12 16:25:49.181036 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:49.180714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6snlj" event={"ID":"6b0dfc77-915e-424b-959b-ab873bf51dbe","Type":"ContainerStarted","Data":"d63711b1a58ad324c004ea8a08ffff3653619bc3be7a51db34ddf3ead80eec3c"} Jul 12 16:25:49.182386 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:49.182349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-th26f" event={"ID":"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18","Type":"ContainerStarted","Data":"8f300117516ad9fbab41113be9a0dac7149de5d9d04c45e2042d84c324c7ca13"} Jul 12 16:25:49.183683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:49.183662 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" event={"ID":"c7fccd46-c827-4d70-827d-70d54b527970","Type":"ContainerStarted","Data":"bc5a9e683f0796ff6cfe48de8b9eedf0be4e015043cdd8311fbc14f68cf97cfc"} Jul 12 16:25:49.201112 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:49.201065 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8hnw8" podStartSLOduration=2.274990892 podStartE2EDuration="8.201046429s" totalImagesPullingTime="5.926055537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:49.199740026 +0000 UTC m=+47.969167920" watchObservedRunningTime="2026-07-12 16:25:49.201046429 +0000 UTC m=+47.970474324" Jul 12 16:25:49.222240 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:49.222167 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6snlj" podStartSLOduration=2.293357471 podStartE2EDuration="8.222146787s" totalImagesPullingTime="5.928789316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:49.220041346 +0000 UTC m=+47.989469243" watchObservedRunningTime="2026-07-12 16:25:49.222146787 +0000 UTC m=+47.991574685" Jul 12 16:25:50.188864 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:50.188828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-th26f" event={"ID":"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18","Type":"ContainerStarted","Data":"f4d2cc3a778172ebd67d3da19546e61be4556bb99a3dd3afbb3878e8aa78be25"} Jul 12 16:25:50.190507 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:50.190475 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" event={"ID":"c7fccd46-c827-4d70-827d-70d54b527970","Type":"ContainerStarted","Data":"49b619d05f67bf9cbde50cfe752566e2c0367995a586a3b20df88b87862833d6"} Jul 12 16:25:50.190894 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:50.190872 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" Jul 12 16:25:50.196488 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:50.196469 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" Jul 12 16:25:50.209071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:50.209032 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g5s49" podStartSLOduration=4.026310498 podStartE2EDuration="5.209020484s" totalImagesPullingTime="1.182709986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:50.20798031 +0000 UTC m=+48.977408208" watchObservedRunningTime="2026-07-12 16:25:50.209020484 +0000 UTC m=+48.978448382" Jul 12 16:25:51.046209 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.046164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:51.145407 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.145370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:51.249769 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.249732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:51.263585 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.263556 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8ld92"] Jul 12 16:25:51.266715 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.266694 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.271420 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.271368 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 12 16:25:51.272747 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.272602 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 12 16:25:51.272747 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.272622 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 12 16:25:51.272747 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.272632 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g7zgt\"" Jul 12 16:25:51.272747 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.272673 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 12 16:25:51.272747 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.272716 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 12 16:25:51.281980 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.281920 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8ld92"] Jul 12 16:25:51.357441 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.357372 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:51.375422 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.375394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbfj\" (UniqueName: \"kubernetes.io/projected/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-kube-api-access-cgbfj\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.375591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.375465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.375591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.375528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.375591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.375555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.476869 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.476831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbfj\" (UniqueName: \"kubernetes.io/projected/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-kube-api-access-cgbfj\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.477097 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.476933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.477097 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.477020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.477097 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:51.477046 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 12 16:25:51.477281 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:25:51.477133 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-tls podName:b79d1dd6-8d16-4388-9ccc-2c896ac5487f nodeName:}" failed. No retries permitted until 2026-07-12 16:25:51.977109828 +0000 UTC m=+50.746537721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-tls") pod "prometheus-operator-64785b6675-8ld92" (UID: "b79d1dd6-8d16-4388-9ccc-2c896ac5487f") : secret "prometheus-operator-tls" not found Jul 12 16:25:51.477281 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.477051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.477828 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.477794 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.479561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.479531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.508622 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.508566 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbfj\" (UniqueName: \"kubernetes.io/projected/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-kube-api-access-cgbfj\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.853698 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.853667 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:51.981507 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.981483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:51.983539 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:51.983519 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b79d1dd6-8d16-4388-9ccc-2c896ac5487f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-8ld92\" (UID: \"b79d1dd6-8d16-4388-9ccc-2c896ac5487f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:52.176817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:52.176788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" Jul 12 16:25:52.199564 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:52.199530 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-th26f" event={"ID":"774f43ee-8efb-4b23-9ffc-5d6ee08f1d18","Type":"ContainerStarted","Data":"75931db95485651c23279c00b5322320e989bd42304e8d0951407b56b17f5cfb"} Jul 12 16:25:52.222415 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:52.222365 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-th26f" podStartSLOduration=1.609738868 podStartE2EDuration="11.222347604s" totalImagesPullingTime="9.612608736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:52.221803188 +0000 UTC m=+50.991231116" watchObservedRunningTime="2026-07-12 16:25:52.222347604 +0000 UTC m=+50.991775499" Jul 12 16:25:52.301424 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:52.301394 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-8ld92"] Jul 12 16:25:52.305244 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:25:52.305161 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79d1dd6_8d16_4388_9ccc_2c896ac5487f.slice/crio-5925aa039336d4e2af133fc2fea9babb9e7027d500a1a62fe8c772f1fe2c2a31 WatchSource:0}: Error finding container 5925aa039336d4e2af133fc2fea9babb9e7027d500a1a62fe8c772f1fe2c2a31: Status 404 returned error can't find the container with id 5925aa039336d4e2af133fc2fea9babb9e7027d500a1a62fe8c772f1fe2c2a31 Jul 12 16:25:53.206397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:53.206358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" event={"ID":"b79d1dd6-8d16-4388-9ccc-2c896ac5487f","Type":"ContainerStarted","Data":"5925aa039336d4e2af133fc2fea9babb9e7027d500a1a62fe8c772f1fe2c2a31"} Jul 12 16:25:53.801305 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:53.801219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:54.211818 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:54.211782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" event={"ID":"b79d1dd6-8d16-4388-9ccc-2c896ac5487f","Type":"ContainerStarted","Data":"d1f8ff13fd4d3e4025fc0814fae6008506ae5141f0b2b8eda3320a89b59f541f"} Jul 12 16:25:54.212014 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:54.211842 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" event={"ID":"b79d1dd6-8d16-4388-9ccc-2c896ac5487f","Type":"ContainerStarted","Data":"f4a7c8a57f8298a0e5185bc1d28997f9f165368700342c57c9b92ac2d6add4f5"} Jul 12 16:25:54.229358 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:54.229310 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-8ld92" podStartSLOduration=2.085877099 podStartE2EDuration="3.229297139s" totalImagesPullingTime="1.14342004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:25:54.228102078 +0000 UTC m=+52.997529973" watchObservedRunningTime="2026-07-12 16:25:54.229297139 +0000 UTC m=+52.998725034" Jul 12 16:25:56.354882 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:56.354820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:56.758943 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:56.758903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:57.086888 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:57.086797 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:57.186883 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:57.186836 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:57.261009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:57.260972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:25:59.193349 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:25:59.193318 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6snlj" Jul 12 16:26:00.202644 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.202608 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:00.288606 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.288575 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-8d645"] Jul 12 16:26:00.296611 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.296588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.297308 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.297284 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:00.299533 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.299514 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 12 16:26:00.299585 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.299562 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 12 16:26:00.299585 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.299572 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xxxb7\"" Jul 12 16:26:00.307940 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.307920 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-8d645"] Jul 12 16:26:00.340013 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.339987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.340142 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.340030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b281d074-ea80-499f-9b85-27da239f3c69-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.340142 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.340050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.340142 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.340094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqbx\" (UniqueName: \"kubernetes.io/projected/b281d074-ea80-499f-9b85-27da239f3c69-kube-api-access-qrqbx\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.377540 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.377511 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:00.398257 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.398235 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-glkh9"] Jul 12 16:26:00.400433 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.400408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.402830 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.402783 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 12 16:26:00.403063 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.403047 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 12 16:26:00.403136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.403099 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 12 16:26:00.403214 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.403134 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-frj9r\"" Jul 12 16:26:00.441049 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2jt\" (UniqueName: \"kubernetes.io/projected/98c3b438-740e-48e7-950e-ce0ad2ada902-kube-api-access-qd2jt\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441051 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-textfile\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b281d074-ea80-499f-9b85-27da239f3c69-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.441158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-wtmp\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441123 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-accelerators-collector-config\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.441158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-root\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-sys\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98c3b438-740e-48e7-950e-ce0ad2ada902-metrics-client-ca\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.441487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqbx\" (UniqueName: \"kubernetes.io/projected/b281d074-ea80-499f-9b85-27da239f3c69-kube-api-access-qrqbx\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.441487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-tls\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.441487 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:26:00.441487 2575 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 12 16:26:00.441830 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:26:00.441542 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-tls podName:b281d074-ea80-499f-9b85-27da239f3c69 nodeName:}" failed. No retries permitted until 2026-07-12 16:26:00.94152921 +0000 UTC m=+59.710957083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-8d645" (UID: "b281d074-ea80-499f-9b85-27da239f3c69") : secret "openshift-state-metrics-tls" not found Jul 12 16:26:00.441830 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.441818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b281d074-ea80-499f-9b85-27da239f3c69-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.443461 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.443439 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.452838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.452788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqbx\" (UniqueName: \"kubernetes.io/projected/b281d074-ea80-499f-9b85-27da239f3c69-kube-api-access-qrqbx\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.467890 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.467865 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:00.542619 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2jt\" (UniqueName: \"kubernetes.io/projected/98c3b438-740e-48e7-950e-ce0ad2ada902-kube-api-access-qd2jt\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.542619 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-textfile\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.542827 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-wtmp\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.542827 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-accelerators-collector-config\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.542827 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542789 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-wtmp\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.542827 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-root\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-sys\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-root\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98c3b438-740e-48e7-950e-ce0ad2ada902-metrics-client-ca\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-textfile\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98c3b438-740e-48e7-950e-ce0ad2ada902-sys\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.542985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.543009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-tls\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543282 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:26:00.543117 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 12 16:26:00.543282 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:26:00.543176 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-tls podName:98c3b438-740e-48e7-950e-ce0ad2ada902 nodeName:}" failed. No retries permitted until 2026-07-12 16:26:01.043158932 +0000 UTC m=+59.812586808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-tls") pod "node-exporter-glkh9" (UID: "98c3b438-740e-48e7-950e-ce0ad2ada902") : secret "node-exporter-tls" not found Jul 12 16:26:00.543282 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.543251 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-accelerators-collector-config\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.543378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.543363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98c3b438-740e-48e7-950e-ce0ad2ada902-metrics-client-ca\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.545144 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.545115 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.551575 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.551548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2jt\" (UniqueName: \"kubernetes.io/projected/98c3b438-740e-48e7-950e-ce0ad2ada902-kube-api-access-qd2jt\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:00.623380 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.623347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:00.695095 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.695059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:00.811103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.811021 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:00.914564 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.914534 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:00.945877 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.945846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:00.948387 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:00.948367 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b281d074-ea80-499f-9b85-27da239f3c69-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-8d645\" (UID: \"b281d074-ea80-499f-9b85-27da239f3c69\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:01.046723 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.046680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-tls\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:01.048904 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.048878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98c3b438-740e-48e7-950e-ce0ad2ada902-node-exporter-tls\") pod \"node-exporter-glkh9\" (UID: \"98c3b438-740e-48e7-950e-ce0ad2ada902\") " pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:01.089312 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.089238 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:01.204882 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.204848 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" Jul 12 16:26:01.308480 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.308450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-glkh9" Jul 12 16:26:01.315366 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.315342 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:01.318124 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:01.318102 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c3b438_740e_48e7_950e_ce0ad2ada902.slice/crio-0c5d6acd8ca70c3771d6761a576d7e02da63733e700be9ab4f55e213e5d28437 WatchSource:0}: Error finding container 0c5d6acd8ca70c3771d6761a576d7e02da63733e700be9ab4f55e213e5d28437: Status 404 returned error can't find the container with id 0c5d6acd8ca70c3771d6761a576d7e02da63733e700be9ab4f55e213e5d28437 Jul 12 16:26:01.335809 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.335786 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-8d645"] Jul 12 16:26:01.339483 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:01.339441 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb281d074_ea80_499f_9b85_27da239f3c69.slice/crio-1cb18faf6c1c4d513549be1e4beed948d1a8af9b2505a4007da757861e366ee6 WatchSource:0}: Error finding container 1cb18faf6c1c4d513549be1e4beed948d1a8af9b2505a4007da757861e366ee6: Status 404 returned error can't find the container with id 1cb18faf6c1c4d513549be1e4beed948d1a8af9b2505a4007da757861e366ee6 Jul 12 16:26:01.417224 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.417165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:01.546966 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.546910 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:01.668940 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:01.668846 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:02.237554 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:02.237509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-glkh9" event={"ID":"98c3b438-740e-48e7-950e-ce0ad2ada902","Type":"ContainerStarted","Data":"0c5d6acd8ca70c3771d6761a576d7e02da63733e700be9ab4f55e213e5d28437"} Jul 12 16:26:02.239653 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:02.239619 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" event={"ID":"b281d074-ea80-499f-9b85-27da239f3c69","Type":"ContainerStarted","Data":"563edfc63efbcf77dc968e7f4628fa022af4e3d446fe4683db3e760772c606e9"} Jul 12 16:26:02.239763 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:02.239677 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" event={"ID":"b281d074-ea80-499f-9b85-27da239f3c69","Type":"ContainerStarted","Data":"d17d680638df313d14480ce4e6bd70af697c0c3803321665e78da060dd743aaf"} Jul 12 16:26:02.239763 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:02.239705 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" event={"ID":"b281d074-ea80-499f-9b85-27da239f3c69","Type":"ContainerStarted","Data":"1cb18faf6c1c4d513549be1e4beed948d1a8af9b2505a4007da757861e366ee6"} Jul 12 16:26:02.325340 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:02.325310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:03.136907 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.136880 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zxb5k" Jul 12 16:26:03.244601 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.244565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" event={"ID":"b281d074-ea80-499f-9b85-27da239f3c69","Type":"ContainerStarted","Data":"31fbb72aa46e23778f2b38e23fda655c7af3cecbe07cf64b1e4a8022976f01b9"} Jul 12 16:26:03.246059 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.246034 2575 generic.go:356] "Generic (PLEG): container finished" podID="98c3b438-740e-48e7-950e-ce0ad2ada902" containerID="d5a7bc0ed1f6634d3c1f61e4ce0608eeff64a438995f0705daa14c9bd462b734" exitCode=0 Jul 12 16:26:03.246141 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.246089 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-glkh9" event={"ID":"98c3b438-740e-48e7-950e-ce0ad2ada902","Type":"ContainerDied","Data":"d5a7bc0ed1f6634d3c1f61e4ce0608eeff64a438995f0705daa14c9bd462b734"} Jul 12 16:26:03.269303 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.269225 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8d645" podStartSLOduration=2.346925979 podStartE2EDuration="3.269180775s" totalImagesPullingTime="922.254796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:03.26799403 +0000 UTC m=+62.037421917" watchObservedRunningTime="2026-07-12 16:26:03.269180775 +0000 UTC m=+62.038608668" Jul 12 16:26:03.323335 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.323310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:03.451655 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.451606 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:03.567357 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.567319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:03.662734 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:03.662710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:04.173816 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.173774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:04.251357 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.251324 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-glkh9" event={"ID":"98c3b438-740e-48e7-950e-ce0ad2ada902","Type":"ContainerStarted","Data":"2ee8b87716a7705bad33441e67a31bf8a4da0053924d5a5dbc90ca99af8efa97"} Jul 12 16:26:04.251710 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.251377 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-glkh9" event={"ID":"98c3b438-740e-48e7-950e-ce0ad2ada902","Type":"ContainerStarted","Data":"b737f3aa641094ad35c08323a3ebd70a97d4012f2db080babdcd70a4af656949"} Jul 12 16:26:04.282663 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.282622 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:04.466717 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.466689 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:04.691372 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.691324 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-glkh9" podStartSLOduration=3.567794225 podStartE2EDuration="4.6913048s" totalImagesPullingTime="1.055801721s" totalInitContainerRuntime="67.708854ms" isStatefulPod=true podCreationTimestamp="2026-07-12 16:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:04.280915016 +0000 UTC m=+63.050342911" watchObservedRunningTime="2026-07-12 16:26:04.6913048 +0000 UTC m=+63.460732695" Jul 12 16:26:04.691594 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.691581 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7ccb4cc598-7crw7"] Jul 12 16:26:04.693610 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.693591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.696818 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.696800 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3sfsi8s915rap\"" Jul 12 16:26:04.696974 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.696947 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 12 16:26:04.697747 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.697729 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mhvtp\"" Jul 12 16:26:04.697841 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.697784 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 12 16:26:04.697841 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.697783 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 12 16:26:04.697948 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.697842 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 12 16:26:04.712374 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.712350 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ccb4cc598-7crw7"] Jul 12 16:26:04.776589 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.776531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb12230-4caa-4739-88b6-16cdb37985fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.776589 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.776562 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfb12230-4caa-4739-88b6-16cdb37985fb-audit-log\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.776589 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.776586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-client-ca-bundle\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.776755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.776645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-secret-metrics-server-client-certs\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.776755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.776668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-secret-metrics-server-tls\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.776755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.776686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfb12230-4caa-4739-88b6-16cdb37985fb-metrics-server-audit-profiles\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.776755 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.776705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrs4q\" (UniqueName: \"kubernetes.io/projected/cfb12230-4caa-4739-88b6-16cdb37985fb-kube-api-access-xrs4q\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.877278 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.877229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb12230-4caa-4739-88b6-16cdb37985fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.877278 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.877279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfb12230-4caa-4739-88b6-16cdb37985fb-audit-log\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.877459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.877395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-client-ca-bundle\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.877459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.877448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-secret-metrics-server-client-certs\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.877522 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.877482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-secret-metrics-server-tls\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.877522 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.877510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfb12230-4caa-4739-88b6-16cdb37985fb-metrics-server-audit-profiles\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.877665 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.877642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrs4q\" (UniqueName: \"kubernetes.io/projected/cfb12230-4caa-4739-88b6-16cdb37985fb-kube-api-access-xrs4q\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.877748 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.877704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfb12230-4caa-4739-88b6-16cdb37985fb-audit-log\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.878122 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.878095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfb12230-4caa-4739-88b6-16cdb37985fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.878661 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.878635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfb12230-4caa-4739-88b6-16cdb37985fb-metrics-server-audit-profiles\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.879920 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.879891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-client-ca-bundle\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.879990 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.879977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-secret-metrics-server-client-certs\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.880038 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.880007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfb12230-4caa-4739-88b6-16cdb37985fb-secret-metrics-server-tls\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:04.887108 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:04.887086 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrs4q\" (UniqueName: \"kubernetes.io/projected/cfb12230-4caa-4739-88b6-16cdb37985fb-kube-api-access-xrs4q\") pod \"metrics-server-7ccb4cc598-7crw7\" (UID: \"cfb12230-4caa-4739-88b6-16cdb37985fb\") " pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:05.001660 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.001613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:05.133330 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.133302 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ccb4cc598-7crw7"] Jul 12 16:26:05.136019 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:05.135980 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb12230_4caa_4739_88b6_16cdb37985fb.slice/crio-3300406666dbe9d1e27949d8b5e6710a6a60d6b818aaf8b956943fa46fb515ea WatchSource:0}: Error finding container 3300406666dbe9d1e27949d8b5e6710a6a60d6b818aaf8b956943fa46fb515ea: Status 404 returned error can't find the container with id 3300406666dbe9d1e27949d8b5e6710a6a60d6b818aaf8b956943fa46fb515ea Jul 12 16:26:05.209971 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.209936 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:05.210123 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.210107 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x"] Jul 12 16:26:05.214358 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.214339 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:05.217293 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.217274 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 12 16:26:05.217411 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.217393 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lz8cx\"" Jul 12 16:26:05.226389 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.226368 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x"] Jul 12 16:26:05.255745 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.255712 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" event={"ID":"cfb12230-4caa-4739-88b6-16cdb37985fb","Type":"ContainerStarted","Data":"3300406666dbe9d1e27949d8b5e6710a6a60d6b818aaf8b956943fa46fb515ea"} Jul 12 16:26:05.271021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.270995 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-87fc84f74-kcb7x"] Jul 12 16:26:05.281416 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.281392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a6b5ac1-015a-4aa3-955e-fb2341b81ead-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-bj27x\" (UID: \"0a6b5ac1-015a-4aa3-955e-fb2341b81ead\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:05.281580 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.281561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5rdk\" (UniqueName: \"kubernetes.io/projected/0a6b5ac1-015a-4aa3-955e-fb2341b81ead-kube-api-access-j5rdk\") pod \"monitoring-plugin-5f68df7f58-bj27x\" (UID: \"0a6b5ac1-015a-4aa3-955e-fb2341b81ead\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:05.292415 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.292395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87fc84f74-kcb7x"] Jul 12 16:26:05.292493 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.292485 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.296049 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.295867 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 12 16:26:05.296561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.296434 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 12 16:26:05.296561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.296488 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 12 16:26:05.296561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.296554 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 12 16:26:05.296866 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.296788 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 12 16:26:05.296975 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.296950 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 12 16:26:05.297555 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.297509 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p55m4\"" Jul 12 16:26:05.297654 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.297577 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 12 16:26:05.304663 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.304645 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 12 16:26:05.321785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.321763 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:05.382643 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.382620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-trusted-ca-bundle\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.382793 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.382650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-console-config\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.382793 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.382669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhc47\" (UniqueName: \"kubernetes.io/projected/2ce45a3d-a334-4371-84bb-303231d63a22-kube-api-access-bhc47\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.382793 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.382709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5rdk\" (UniqueName: \"kubernetes.io/projected/0a6b5ac1-015a-4aa3-955e-fb2341b81ead-kube-api-access-j5rdk\") pod \"monitoring-plugin-5f68df7f58-bj27x\" (UID: \"0a6b5ac1-015a-4aa3-955e-fb2341b81ead\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:05.382956 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.382816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-serving-cert\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.382956 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.382839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a6b5ac1-015a-4aa3-955e-fb2341b81ead-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-bj27x\" (UID: \"0a6b5ac1-015a-4aa3-955e-fb2341b81ead\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:05.382956 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.382862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-oauth-config\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.382956 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.382880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-service-ca\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.383093 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.383017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-oauth-serving-cert\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.385103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.385078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a6b5ac1-015a-4aa3-955e-fb2341b81ead-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-bj27x\" (UID: \"0a6b5ac1-015a-4aa3-955e-fb2341b81ead\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:05.391798 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.391780 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5rdk\" (UniqueName: \"kubernetes.io/projected/0a6b5ac1-015a-4aa3-955e-fb2341b81ead-kube-api-access-j5rdk\") pod \"monitoring-plugin-5f68df7f58-bj27x\" (UID: \"0a6b5ac1-015a-4aa3-955e-fb2341b81ead\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:05.413966 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.413941 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:05.483800 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.483775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-trusted-ca-bundle\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.483800 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.483801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-console-config\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.483940 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.483822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhc47\" (UniqueName: \"kubernetes.io/projected/2ce45a3d-a334-4371-84bb-303231d63a22-kube-api-access-bhc47\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.484002 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.483979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-serving-cert\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.484052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.484024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-oauth-config\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.484102 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.484048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-service-ca\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.484161 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.484146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-oauth-serving-cert\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.484569 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.484543 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-console-config\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.484687 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.484667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-service-ca\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.484739 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.484704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-trusted-ca-bundle\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.484809 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.484795 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-oauth-serving-cert\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.486370 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.486344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-serving-cert\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.486444 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.486391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-oauth-config\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.504963 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.504943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhc47\" (UniqueName: \"kubernetes.io/projected/2ce45a3d-a334-4371-84bb-303231d63a22-kube-api-access-bhc47\") pod \"console-87fc84f74-kcb7x\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.523311 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.523281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:05.594776 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.594750 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq"] Jul 12 16:26:05.603971 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.603900 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:05.622828 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.622802 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq"] Jul 12 16:26:05.622940 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.622923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.625385 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.625367 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kq2qj\"" Jul 12 16:26:05.625477 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.625430 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 12 16:26:05.625477 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.625430 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 12 16:26:05.625682 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.625520 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 12 16:26:05.625820 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.625807 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 12 16:26:05.626362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.626341 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 12 16:26:05.631750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.631730 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 12 16:26:05.686006 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.685967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-serving-certs-ca-bundle\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.686149 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.686028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-secret-telemeter-client\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.686149 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.686066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.686149 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.686111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-telemeter-client-tls\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.686149 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.686126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-federate-client-tls\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.686357 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.686163 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j44wq\" (UniqueName: \"kubernetes.io/projected/eb71236f-3c31-4c39-b118-481c18e8e516-kube-api-access-j44wq\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.686357 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.686257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-metrics-client-ca\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.686357 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.686294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-telemeter-trusted-ca-bundle\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.697594 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.697569 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x"] Jul 12 16:26:05.700701 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:05.700641 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6b5ac1_015a_4aa3_955e_fb2341b81ead.slice/crio-3f6d8767bccaf4b738ad1d764d70b25bb31cab4eb7fc229b059993b01c60cf0c WatchSource:0}: Error finding container 3f6d8767bccaf4b738ad1d764d70b25bb31cab4eb7fc229b059993b01c60cf0c: Status 404 returned error can't find the container with id 3f6d8767bccaf4b738ad1d764d70b25bb31cab4eb7fc229b059993b01c60cf0c Jul 12 16:26:05.779164 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.779085 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87fc84f74-kcb7x"] Jul 12 16:26:05.782227 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:05.782200 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce45a3d_a334_4371_84bb_303231d63a22.slice/crio-32e0bb6c081a13eb79c7ac3b8415a342ca9579637c5ee76b65d88089f3d6a00e WatchSource:0}: Error finding container 32e0bb6c081a13eb79c7ac3b8415a342ca9579637c5ee76b65d88089f3d6a00e: Status 404 returned error can't find the container with id 32e0bb6c081a13eb79c7ac3b8415a342ca9579637c5ee76b65d88089f3d6a00e Jul 12 16:26:05.787340 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.787316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.787445 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.787352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-telemeter-client-tls\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.787445 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.787373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-federate-client-tls\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.787557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.787464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j44wq\" (UniqueName: \"kubernetes.io/projected/eb71236f-3c31-4c39-b118-481c18e8e516-kube-api-access-j44wq\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.787557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.787511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-metrics-client-ca\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.787557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.787543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-telemeter-trusted-ca-bundle\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.787729 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.787698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-serving-certs-ca-bundle\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.787811 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.787784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-secret-telemeter-client\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.788548 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.788469 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-metrics-client-ca\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.788548 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.788504 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-serving-certs-ca-bundle\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.788685 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.788589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb71236f-3c31-4c39-b118-481c18e8e516-telemeter-trusted-ca-bundle\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.790374 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.790351 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.790668 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.790623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-secret-telemeter-client\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.790938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.790920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-federate-client-tls\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.791871 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.791853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb71236f-3c31-4c39-b118-481c18e8e516-telemeter-client-tls\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.795980 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.795958 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j44wq\" (UniqueName: \"kubernetes.io/projected/eb71236f-3c31-4c39-b118-481c18e8e516-kube-api-access-j44wq\") pod \"telemeter-client-644bcfc9c8-c4qxq\" (UID: \"eb71236f-3c31-4c39-b118-481c18e8e516\") " pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:05.932977 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:05.932943 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" Jul 12 16:26:06.094134 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.093916 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq"] Jul 12 16:26:06.096319 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:06.096285 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb71236f_3c31_4c39_b118_481c18e8e516.slice/crio-b1ac981b1a06ee18353c186b82c1384773fd725fdea1f77e0c37352fc98d8da7 WatchSource:0}: Error finding container b1ac981b1a06ee18353c186b82c1384773fd725fdea1f77e0c37352fc98d8da7: Status 404 returned error can't find the container with id b1ac981b1a06ee18353c186b82c1384773fd725fdea1f77e0c37352fc98d8da7 Jul 12 16:26:06.262705 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.262617 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" event={"ID":"eb71236f-3c31-4c39-b118-481c18e8e516","Type":"ContainerStarted","Data":"b1ac981b1a06ee18353c186b82c1384773fd725fdea1f77e0c37352fc98d8da7"} Jul 12 16:26:06.264368 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.264327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87fc84f74-kcb7x" event={"ID":"2ce45a3d-a334-4371-84bb-303231d63a22","Type":"ContainerStarted","Data":"32e0bb6c081a13eb79c7ac3b8415a342ca9579637c5ee76b65d88089f3d6a00e"} Jul 12 16:26:06.265818 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.265766 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" event={"ID":"0a6b5ac1-015a-4aa3-955e-fb2341b81ead","Type":"ContainerStarted","Data":"3f6d8767bccaf4b738ad1d764d70b25bb31cab4eb7fc229b059993b01c60cf0c"} Jul 12 16:26:06.495699 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.495650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:26:06.495874 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.495752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:26:06.498295 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.498160 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 16:26:06.498490 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.498411 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 16:26:06.509140 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.509095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97442fa3-b0da-4f2d-a0a7-b5682b9c2402-original-pull-secret\") pod \"global-pull-secret-syncer-4z46t\" (UID: \"97442fa3-b0da-4f2d-a0a7-b5682b9c2402\") " pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:26:06.509630 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.509572 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207a370a-c6ee-42b3-a858-1f9ec07a7191-metrics-certs\") pod \"network-metrics-daemon-hpv4n\" (UID: \"207a370a-c6ee-42b3-a858-1f9ec07a7191\") " pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:26:06.597229 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.597121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:26:06.600061 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.599842 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 16:26:06.611496 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.611282 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 16:26:06.623073 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.623041 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8gm\" (UniqueName: \"kubernetes.io/projected/7e75bd51-6735-4f43-afc5-de7e1d45e98b-kube-api-access-cs8gm\") pod \"network-check-target-blsjj\" (UID: \"7e75bd51-6735-4f43-afc5-de7e1d45e98b\") " pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:26:06.625508 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.625472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:06.687815 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.687776 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:06.780296 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.780244 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z46t" Jul 12 16:26:06.787900 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.787868 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xb45\"" Jul 12 16:26:06.791936 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.791905 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kpl6g\"" Jul 12 16:26:06.796028 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.796008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpv4n" Jul 12 16:26:06.800730 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.800708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:26:06.813452 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:06.813411 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:07.611315 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:07.611157 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4z46t"] Jul 12 16:26:07.617673 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:07.617640 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97442fa3_b0da_4f2d_a0a7_b5682b9c2402.slice/crio-8c0662d927444f16a007bf77285fe236a4201a721609b82eb2492d9a06dbe515 WatchSource:0}: Error finding container 8c0662d927444f16a007bf77285fe236a4201a721609b82eb2492d9a06dbe515: Status 404 returned error can't find the container with id 8c0662d927444f16a007bf77285fe236a4201a721609b82eb2492d9a06dbe515 Jul 12 16:26:07.628596 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:07.628376 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hpv4n"] Jul 12 16:26:07.632419 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:07.632389 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207a370a_c6ee_42b3_a858_1f9ec07a7191.slice/crio-26d8a57bf4226b007de108d9d2c9286a8a3c30b44e6bafdf67cb2d2b66c9f8cd WatchSource:0}: Error finding container 26d8a57bf4226b007de108d9d2c9286a8a3c30b44e6bafdf67cb2d2b66c9f8cd: Status 404 returned error can't find the container with id 26d8a57bf4226b007de108d9d2c9286a8a3c30b44e6bafdf67cb2d2b66c9f8cd Jul 12 16:26:07.651686 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:07.651648 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-blsjj"] Jul 12 16:26:07.656683 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:07.656422 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e75bd51_6735_4f43_afc5_de7e1d45e98b.slice/crio-cd5ce2c89a985da8558a2d81f0e1c38acaa3cc2d73bf02f530d69f6963caf2a1 WatchSource:0}: Error finding container cd5ce2c89a985da8558a2d81f0e1c38acaa3cc2d73bf02f530d69f6963caf2a1: Status 404 returned error can't find the container with id cd5ce2c89a985da8558a2d81f0e1c38acaa3cc2d73bf02f530d69f6963caf2a1 Jul 12 16:26:08.019413 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.019353 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:08.080184 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.080106 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:08.276552 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.276466 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" event={"ID":"cfb12230-4caa-4739-88b6-16cdb37985fb","Type":"ContainerStarted","Data":"fd6c7a2fcf2c174b24ca962c4edb3491f0f01179b92ee4e43be38b245ae36b74"} Jul 12 16:26:08.277788 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.277763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-blsjj" event={"ID":"7e75bd51-6735-4f43-afc5-de7e1d45e98b","Type":"ContainerStarted","Data":"cd5ce2c89a985da8558a2d81f0e1c38acaa3cc2d73bf02f530d69f6963caf2a1"} Jul 12 16:26:08.278985 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.278962 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4z46t" event={"ID":"97442fa3-b0da-4f2d-a0a7-b5682b9c2402","Type":"ContainerStarted","Data":"8c0662d927444f16a007bf77285fe236a4201a721609b82eb2492d9a06dbe515"} Jul 12 16:26:08.280578 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.280551 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" event={"ID":"0a6b5ac1-015a-4aa3-955e-fb2341b81ead","Type":"ContainerStarted","Data":"91cba4ba302181081eec28e4667b44c2df65513e40864736a7a88991c93affbf"} Jul 12 16:26:08.280965 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.280938 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:08.282107 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.282063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpv4n" event={"ID":"207a370a-c6ee-42b3-a858-1f9ec07a7191","Type":"ContainerStarted","Data":"26d8a57bf4226b007de108d9d2c9286a8a3c30b44e6bafdf67cb2d2b66c9f8cd"} Jul 12 16:26:08.286634 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.286615 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" Jul 12 16:26:08.297159 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.297101 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" podStartSLOduration=2.001010921 podStartE2EDuration="4.29708668s" totalImagesPullingTime="2.296075759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:08.296719321 +0000 UTC m=+67.066147217" watchObservedRunningTime="2026-07-12 16:26:08.29708668 +0000 UTC m=+67.066514575" Jul 12 16:26:08.369240 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:08.369175 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-bj27x" podStartSLOduration=1.631038252 podStartE2EDuration="3.369153831s" totalImagesPullingTime="1.738115579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:08.368470972 +0000 UTC m=+67.137898867" watchObservedRunningTime="2026-07-12 16:26:08.369153831 +0000 UTC m=+67.138581726" Jul 12 16:26:09.295739 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:09.295681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:09.362183 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:09.362146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:10.293091 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:10.293053 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87fc84f74-kcb7x" event={"ID":"2ce45a3d-a334-4371-84bb-303231d63a22","Type":"ContainerStarted","Data":"52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a"} Jul 12 16:26:10.296466 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:10.296438 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" event={"ID":"eb71236f-3c31-4c39-b118-481c18e8e516","Type":"ContainerStarted","Data":"6a3e1bacaa8eb3dc1570429b7abe1a989fb0ce39534e9816d7e8f54256553131"} Jul 12 16:26:11.303529 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.303048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" event={"ID":"eb71236f-3c31-4c39-b118-481c18e8e516","Type":"ContainerStarted","Data":"435c52c2b606715951136b0f0bff03c1a1d072f10ce47c748dc9d7adc4d94d84"} Jul 12 16:26:11.303529 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.303127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" event={"ID":"eb71236f-3c31-4c39-b118-481c18e8e516","Type":"ContainerStarted","Data":"c226ea599072ffe99a1021b78d6831952f347257d028b0a5960e084f0f37ab11"} Jul 12 16:26:11.305928 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.305876 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpv4n" event={"ID":"207a370a-c6ee-42b3-a858-1f9ec07a7191","Type":"ContainerStarted","Data":"21d07700b8700168eed9b42cc75845ba5a1bbf34b37e8c0e8cd27ecad4a2f32d"} Jul 12 16:26:11.306074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.305942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpv4n" event={"ID":"207a370a-c6ee-42b3-a858-1f9ec07a7191","Type":"ContainerStarted","Data":"b7eccf647146181cd3b160da8c88ba287f469e5706d3be6a838d63599e9a1ada"} Jul 12 16:26:11.329991 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.329887 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-644bcfc9c8-c4qxq" podStartSLOduration=1.5042850859999999 podStartE2EDuration="6.329866294s" totalImagesPullingTime="4.825581208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:26:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:11.328236255 +0000 UTC m=+70.097664153" watchObservedRunningTime="2026-07-12 16:26:11.329866294 +0000 UTC m=+70.099294190" Jul 12 16:26:11.330619 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.330582 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-87fc84f74-kcb7x" podStartSLOduration=2.6118395210000003 podStartE2EDuration="6.330570872s" totalImagesPullingTime="3.718731351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:10.312003885 +0000 UTC m=+69.081431793" watchObservedRunningTime="2026-07-12 16:26:11.330570872 +0000 UTC m=+70.099998766" Jul 12 16:26:11.346936 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.346559 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hpv4n" podStartSLOduration=66.751386024 podStartE2EDuration="1m9.346543959s" totalImagesPullingTime="2.595157935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:11.345077706 +0000 UTC m=+70.114505601" watchObservedRunningTime="2026-07-12 16:26:11.346543959 +0000 UTC m=+70.115971855" Jul 12 16:26:11.482447 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.482404 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:11.588029 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:11.587946 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:12.054068 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.054035 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87fc84f74-kcb7x"] Jul 12 16:26:12.089957 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.089924 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b69f5d8cc-bt6s5"] Jul 12 16:26:12.093619 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.093050 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.103282 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.103227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b69f5d8cc-bt6s5"] Jul 12 16:26:12.148455 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.148427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26wb6\" (UniqueName: \"kubernetes.io/projected/2e2758f0-29d1-4670-8a63-194956985664-kube-api-access-26wb6\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.148632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.148472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-oauth-serving-cert\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.148632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.148500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-service-ca\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.148632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.148526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-serving-cert\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.148632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.148549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-oauth-config\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.148632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.148602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-console-config\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.148878 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.148690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-trusted-ca-bundle\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.252018 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.251986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-trusted-ca-bundle\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.252258 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.252029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26wb6\" (UniqueName: \"kubernetes.io/projected/2e2758f0-29d1-4670-8a63-194956985664-kube-api-access-26wb6\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.252258 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.252051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-oauth-serving-cert\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.252258 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.252075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-service-ca\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.252258 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.252236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-serving-cert\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.252467 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.252286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-oauth-config\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.252467 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.252364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-console-config\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.253033 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.253006 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-trusted-ca-bundle\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.253129 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.253036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-service-ca\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.253129 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.253078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-oauth-serving-cert\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.253280 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.253135 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-console-config\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.255216 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.255162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-oauth-config\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.255606 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.255590 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-serving-cert\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.261308 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.261290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26wb6\" (UniqueName: \"kubernetes.io/projected/2e2758f0-29d1-4670-8a63-194956985664-kube-api-access-26wb6\") pod \"console-7b69f5d8cc-bt6s5\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.318995 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.318924 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:12.407741 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.407704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:12.415888 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:12.415865 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:13.960794 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:13.960665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b69f5d8cc-bt6s5"] Jul 12 16:26:13.963106 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:26:13.963084 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2758f0_29d1_4670_8a63_194956985664.slice/crio-708e35bd0d6f2ad1f2545b35a655ed9130378c731aafc5691cfc161805cbcaba WatchSource:0}: Error finding container 708e35bd0d6f2ad1f2545b35a655ed9130378c731aafc5691cfc161805cbcaba: Status 404 returned error can't find the container with id 708e35bd0d6f2ad1f2545b35a655ed9130378c731aafc5691cfc161805cbcaba Jul 12 16:26:14.205864 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.205828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:14.307015 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.306980 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:14.318666 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.318630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b69f5d8cc-bt6s5" event={"ID":"2e2758f0-29d1-4670-8a63-194956985664","Type":"ContainerStarted","Data":"41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85"} Jul 12 16:26:14.318806 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.318714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b69f5d8cc-bt6s5" event={"ID":"2e2758f0-29d1-4670-8a63-194956985664","Type":"ContainerStarted","Data":"708e35bd0d6f2ad1f2545b35a655ed9130378c731aafc5691cfc161805cbcaba"} Jul 12 16:26:14.320015 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.319994 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-blsjj" event={"ID":"7e75bd51-6735-4f43-afc5-de7e1d45e98b","Type":"ContainerStarted","Data":"77fbf32b38792856673a6acc3d974cf87cbe5b7ec308ae6a9f3df514974d5e25"} Jul 12 16:26:14.320139 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.320037 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:26:14.321235 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.321217 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4z46t" event={"ID":"97442fa3-b0da-4f2d-a0a7-b5682b9c2402","Type":"ContainerStarted","Data":"516f98f9d64b8af6dcbaa8f34ddf67d4b9c01f34f80f6dd4efe703d87d32b5bc"} Jul 12 16:26:14.340338 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.340292 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b69f5d8cc-bt6s5" podStartSLOduration=2.340273879 podStartE2EDuration="2.340273879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:14.339875051 +0000 UTC m=+73.109303009" watchObservedRunningTime="2026-07-12 16:26:14.340273879 +0000 UTC m=+73.109701773" Jul 12 16:26:14.356986 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.356936 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-blsjj" podStartSLOduration=67.18992569 podStartE2EDuration="1m13.356917204s" totalImagesPullingTime="6.166991514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:14.3561719 +0000 UTC m=+73.125599795" watchObservedRunningTime="2026-07-12 16:26:14.356917204 +0000 UTC m=+73.126345207" Jul 12 16:26:14.378028 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:14.377983 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4z46t" podStartSLOduration=66.161035549 podStartE2EDuration="1m12.37796251s" totalImagesPullingTime="6.216926961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 16:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:26:14.377427553 +0000 UTC m=+73.146855451" watchObservedRunningTime="2026-07-12 16:26:14.37796251 +0000 UTC m=+73.147390405" Jul 12 16:26:15.604709 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:15.604672 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:17.099845 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:17.099810 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:17.207632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:17.207601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:22.408744 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:22.408708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:22.409147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:22.408769 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:22.414042 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:22.414018 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:23.360385 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:23.360281 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:26:23.641952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:23.641875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:23.805851 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:23.805820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:25.002109 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:25.002068 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:25.002522 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:25.002135 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:32.137131 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:32.137092 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:32.200450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:32.200410 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:37.333372 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.333319 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-87fc84f74-kcb7x" podUID="2ce45a3d-a334-4371-84bb-303231d63a22" containerName="console" containerID="cri-o://52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a" gracePeriod=15 Jul 12 16:26:37.567847 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.567824 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:37.656523 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656444 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-trusted-ca-bundle\") pod \"2ce45a3d-a334-4371-84bb-303231d63a22\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " Jul 12 16:26:37.656523 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656486 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhc47\" (UniqueName: \"kubernetes.io/projected/2ce45a3d-a334-4371-84bb-303231d63a22-kube-api-access-bhc47\") pod \"2ce45a3d-a334-4371-84bb-303231d63a22\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " Jul 12 16:26:37.656523 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656526 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-serving-cert\") pod \"2ce45a3d-a334-4371-84bb-303231d63a22\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " Jul 12 16:26:37.656783 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656545 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-oauth-serving-cert\") pod \"2ce45a3d-a334-4371-84bb-303231d63a22\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " Jul 12 16:26:37.656783 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656565 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-service-ca\") pod \"2ce45a3d-a334-4371-84bb-303231d63a22\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " Jul 12 16:26:37.656783 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656593 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-oauth-config\") pod \"2ce45a3d-a334-4371-84bb-303231d63a22\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " Jul 12 16:26:37.656947 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656791 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-console-config\") pod \"2ce45a3d-a334-4371-84bb-303231d63a22\" (UID: \"2ce45a3d-a334-4371-84bb-303231d63a22\") " Jul 12 16:26:37.657001 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656937 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2ce45a3d-a334-4371-84bb-303231d63a22" (UID: "2ce45a3d-a334-4371-84bb-303231d63a22"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:26:37.657054 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.656993 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-service-ca" (OuterVolumeSpecName: "service-ca") pod "2ce45a3d-a334-4371-84bb-303231d63a22" (UID: "2ce45a3d-a334-4371-84bb-303231d63a22"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:26:37.657054 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.657018 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2ce45a3d-a334-4371-84bb-303231d63a22" (UID: "2ce45a3d-a334-4371-84bb-303231d63a22"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:26:37.657236 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.657218 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-trusted-ca-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:26:37.657302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.657241 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-oauth-serving-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:26:37.657302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.657248 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-console-config" (OuterVolumeSpecName: "console-config") pod "2ce45a3d-a334-4371-84bb-303231d63a22" (UID: "2ce45a3d-a334-4371-84bb-303231d63a22"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:26:37.657302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.657255 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-service-ca\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:26:37.658887 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.658865 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce45a3d-a334-4371-84bb-303231d63a22-kube-api-access-bhc47" (OuterVolumeSpecName: "kube-api-access-bhc47") pod "2ce45a3d-a334-4371-84bb-303231d63a22" (UID: "2ce45a3d-a334-4371-84bb-303231d63a22"). InnerVolumeSpecName "kube-api-access-bhc47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:26:37.659225 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.659202 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2ce45a3d-a334-4371-84bb-303231d63a22" (UID: "2ce45a3d-a334-4371-84bb-303231d63a22"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:26:37.659308 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.659287 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2ce45a3d-a334-4371-84bb-303231d63a22" (UID: "2ce45a3d-a334-4371-84bb-303231d63a22"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:26:37.757846 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.757820 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-oauth-config\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:26:37.757846 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.757844 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ce45a3d-a334-4371-84bb-303231d63a22-console-config\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:26:37.757846 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.757854 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhc47\" (UniqueName: \"kubernetes.io/projected/2ce45a3d-a334-4371-84bb-303231d63a22-kube-api-access-bhc47\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:26:37.758054 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:37.757863 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ce45a3d-a334-4371-84bb-303231d63a22-console-serving-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:26:38.416931 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.416893 2575 generic.go:356] "Generic (PLEG): container finished" podID="2ce45a3d-a334-4371-84bb-303231d63a22" containerID="52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a" exitCode=0 Jul 12 16:26:38.417349 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.416954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87fc84f74-kcb7x" Jul 12 16:26:38.417349 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.416959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87fc84f74-kcb7x" event={"ID":"2ce45a3d-a334-4371-84bb-303231d63a22","Type":"ContainerDied","Data":"52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a"} Jul 12 16:26:38.417349 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.417034 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87fc84f74-kcb7x" event={"ID":"2ce45a3d-a334-4371-84bb-303231d63a22","Type":"ContainerDied","Data":"32e0bb6c081a13eb79c7ac3b8415a342ca9579637c5ee76b65d88089f3d6a00e"} Jul 12 16:26:38.417349 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.417061 2575 scope.go:117] "RemoveContainer" containerID="52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a" Jul 12 16:26:38.424856 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.424834 2575 scope.go:117] "RemoveContainer" containerID="52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a" Jul 12 16:26:38.425109 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:26:38.425090 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a\": container with ID starting with 52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a not found: ID does not exist" containerID="52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a" Jul 12 16:26:38.425165 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.425116 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a"} err="failed to get container status \"52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a\": rpc error: code = NotFound desc = could not find container \"52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a\": container with ID starting with 52a81a1029c1c589633f623982e045705c1b119200a502a4ca36524d0ec7c58a not found: ID does not exist" Jul 12 16:26:38.434540 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.434520 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87fc84f74-kcb7x"] Jul 12 16:26:38.440947 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:38.440927 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-87fc84f74-kcb7x"] Jul 12 16:26:39.873572 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:39.873540 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce45a3d-a334-4371-84bb-303231d63a22" path="/var/lib/kubelet/pods/2ce45a3d-a334-4371-84bb-303231d63a22/volumes" Jul 12 16:26:45.007618 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:45.007588 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:45.011273 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:45.011240 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7ccb4cc598-7crw7" Jul 12 16:26:45.327487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:45.327409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-blsjj" Jul 12 16:26:47.140129 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:47.140099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:26:47.240528 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:26:47.240494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:16.629706 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:16.629674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:16.679712 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:16.679684 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:20.930767 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.930731 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55dc84b75c-2tbh6"] Jul 12 16:27:20.931162 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.931015 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce45a3d-a334-4371-84bb-303231d63a22" containerName="console" Jul 12 16:27:20.931162 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.931027 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce45a3d-a334-4371-84bb-303231d63a22" containerName="console" Jul 12 16:27:20.931162 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.931077 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce45a3d-a334-4371-84bb-303231d63a22" containerName="console" Jul 12 16:27:20.933662 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.933639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:20.942955 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.942830 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55dc84b75c-2tbh6"] Jul 12 16:27:20.963999 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.963969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-oauth-serving-cert\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:20.964118 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.964007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-oauth-config\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:20.964118 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.964030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-trusted-ca-bundle\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:20.964118 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.964106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-service-ca\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:20.964245 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.964142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-serving-cert\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:20.964245 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.964172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c264m\" (UniqueName: \"kubernetes.io/projected/7203e0bf-5a0f-45ba-b76e-e006127f91da-kube-api-access-c264m\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:20.964310 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:20.964266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-config\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.014840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.014805 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:21.064780 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.064737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-config\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.064780 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.064776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-oauth-serving-cert\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.064796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-oauth-config\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.064811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-trusted-ca-bundle\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.064909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-service-ca\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.064937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-serving-cert\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.064976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c264m\" (UniqueName: \"kubernetes.io/projected/7203e0bf-5a0f-45ba-b76e-e006127f91da-kube-api-access-c264m\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065576 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.065553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-config\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065669 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.065634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-oauth-serving-cert\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065721 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.065663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-trusted-ca-bundle\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.065758 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.065731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-service-ca\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.067320 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.067298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-oauth-config\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.067447 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.067431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-serving-cert\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.070453 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.070429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:21.072345 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.072325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c264m\" (UniqueName: \"kubernetes.io/projected/7203e0bf-5a0f-45ba-b76e-e006127f91da-kube-api-access-c264m\") pod \"console-55dc84b75c-2tbh6\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.242467 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.242377 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:21.332512 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.329897 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:21.381849 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.381816 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55dc84b75c-2tbh6"] Jul 12 16:27:21.384692 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:27:21.384665 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7203e0bf_5a0f_45ba_b76e_e006127f91da.slice/crio-5035a837634f9e457c37c9905770dde5ef5892e959d57c529dcd3cb431423b0b WatchSource:0}: Error finding container 5035a837634f9e457c37c9905770dde5ef5892e959d57c529dcd3cb431423b0b: Status 404 returned error can't find the container with id 5035a837634f9e457c37c9905770dde5ef5892e959d57c529dcd3cb431423b0b Jul 12 16:27:21.564778 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.564695 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dc84b75c-2tbh6" event={"ID":"7203e0bf-5a0f-45ba-b76e-e006127f91da","Type":"ContainerStarted","Data":"201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac"} Jul 12 16:27:21.564778 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.564747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dc84b75c-2tbh6" event={"ID":"7203e0bf-5a0f-45ba-b76e-e006127f91da","Type":"ContainerStarted","Data":"5035a837634f9e457c37c9905770dde5ef5892e959d57c529dcd3cb431423b0b"} Jul 12 16:27:21.582266 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:21.582217 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55dc84b75c-2tbh6" podStartSLOduration=1.582182391 podStartE2EDuration="1.582182391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:27:21.580805984 +0000 UTC m=+140.350233880" watchObservedRunningTime="2026-07-12 16:27:21.582182391 +0000 UTC m=+140.351610286" Jul 12 16:27:22.119455 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:22.119423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:22.174632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:22.174605 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:28.498170 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:28.498136 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:28.554949 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:28.554917 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:31.243321 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:31.243274 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:31.243724 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:31.243341 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:31.248072 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:31.248049 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:31.604107 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:31.604021 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:27:31.647577 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:31.647549 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b69f5d8cc-bt6s5"] Jul 12 16:27:31.751446 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:31.751412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:31.852509 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:31.852475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:31.905657 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:31.905574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:32.140559 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:32.140528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:32.244428 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:32.244396 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:53.165205 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:53.165166 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:27:56.669249 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:56.669206 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b69f5d8cc-bt6s5" podUID="2e2758f0-29d1-4670-8a63-194956985664" containerName="console" containerID="cri-o://41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85" gracePeriod=15 Jul 12 16:27:56.905251 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:56.905204 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:27:57.035161 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035122 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-oauth-config\") pod \"2e2758f0-29d1-4670-8a63-194956985664\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " Jul 12 16:27:57.035161 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035164 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26wb6\" (UniqueName: \"kubernetes.io/projected/2e2758f0-29d1-4670-8a63-194956985664-kube-api-access-26wb6\") pod \"2e2758f0-29d1-4670-8a63-194956985664\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " Jul 12 16:27:57.035448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035182 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-serving-cert\") pod \"2e2758f0-29d1-4670-8a63-194956985664\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " Jul 12 16:27:57.035448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035235 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-console-config\") pod \"2e2758f0-29d1-4670-8a63-194956985664\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " Jul 12 16:27:57.035448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035254 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-service-ca\") pod \"2e2758f0-29d1-4670-8a63-194956985664\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " Jul 12 16:27:57.035448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035302 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-oauth-serving-cert\") pod \"2e2758f0-29d1-4670-8a63-194956985664\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " Jul 12 16:27:57.035448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035317 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-trusted-ca-bundle\") pod \"2e2758f0-29d1-4670-8a63-194956985664\" (UID: \"2e2758f0-29d1-4670-8a63-194956985664\") " Jul 12 16:27:57.035779 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035744 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-console-config" (OuterVolumeSpecName: "console-config") pod "2e2758f0-29d1-4670-8a63-194956985664" (UID: "2e2758f0-29d1-4670-8a63-194956985664"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:27:57.035855 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035751 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2e2758f0-29d1-4670-8a63-194956985664" (UID: "2e2758f0-29d1-4670-8a63-194956985664"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:27:57.035855 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035757 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-service-ca" (OuterVolumeSpecName: "service-ca") pod "2e2758f0-29d1-4670-8a63-194956985664" (UID: "2e2758f0-29d1-4670-8a63-194956985664"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:27:57.035855 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.035805 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2e2758f0-29d1-4670-8a63-194956985664" (UID: "2e2758f0-29d1-4670-8a63-194956985664"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:27:57.037462 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.037431 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2e2758f0-29d1-4670-8a63-194956985664" (UID: "2e2758f0-29d1-4670-8a63-194956985664"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:27:57.037554 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.037481 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2e2758f0-29d1-4670-8a63-194956985664" (UID: "2e2758f0-29d1-4670-8a63-194956985664"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:27:57.037554 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.037492 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2758f0-29d1-4670-8a63-194956985664-kube-api-access-26wb6" (OuterVolumeSpecName: "kube-api-access-26wb6") pod "2e2758f0-29d1-4670-8a63-194956985664" (UID: "2e2758f0-29d1-4670-8a63-194956985664"). InnerVolumeSpecName "kube-api-access-26wb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:27:57.136611 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.136576 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-oauth-serving-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:27:57.136611 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.136606 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-trusted-ca-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:27:57.136611 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.136616 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-oauth-config\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:27:57.136840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.136626 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26wb6\" (UniqueName: \"kubernetes.io/projected/2e2758f0-29d1-4670-8a63-194956985664-kube-api-access-26wb6\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:27:57.136840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.136635 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2758f0-29d1-4670-8a63-194956985664-console-serving-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:27:57.136840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.136646 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-console-config\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:27:57.136840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.136655 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2758f0-29d1-4670-8a63-194956985664-service-ca\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:27:57.690410 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.690373 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e2758f0-29d1-4670-8a63-194956985664" containerID="41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85" exitCode=0 Jul 12 16:27:57.690785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.690424 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b69f5d8cc-bt6s5" Jul 12 16:27:57.690785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.690431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b69f5d8cc-bt6s5" event={"ID":"2e2758f0-29d1-4670-8a63-194956985664","Type":"ContainerDied","Data":"41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85"} Jul 12 16:27:57.690785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.690481 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b69f5d8cc-bt6s5" event={"ID":"2e2758f0-29d1-4670-8a63-194956985664","Type":"ContainerDied","Data":"708e35bd0d6f2ad1f2545b35a655ed9130378c731aafc5691cfc161805cbcaba"} Jul 12 16:27:57.690785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.690511 2575 scope.go:117] "RemoveContainer" containerID="41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85" Jul 12 16:27:57.698521 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.698498 2575 scope.go:117] "RemoveContainer" containerID="41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85" Jul 12 16:27:57.698751 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:27:57.698735 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85\": container with ID starting with 41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85 not found: ID does not exist" containerID="41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85" Jul 12 16:27:57.698797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.698759 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85"} err="failed to get container status \"41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85\": rpc error: code = NotFound desc = could not find container \"41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85\": container with ID starting with 41e0424505dd510e66a9012501a1c19980b7f52870c868ff056c58ffc66a2f85 not found: ID does not exist" Jul 12 16:27:57.708797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.708769 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b69f5d8cc-bt6s5"] Jul 12 16:27:57.712528 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.712508 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b69f5d8cc-bt6s5"] Jul 12 16:27:57.873724 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:27:57.873692 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e2758f0-29d1-4670-8a63-194956985664" path="/var/lib/kubelet/pods/2e2758f0-29d1-4670-8a63-194956985664/volumes" Jul 12 16:28:15.344361 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.344323 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:15.444955 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.444929 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:15.979963 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.979930 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf"] Jul 12 16:28:15.980222 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.980198 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2758f0-29d1-4670-8a63-194956985664" containerName="console" Jul 12 16:28:15.980222 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.980213 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2758f0-29d1-4670-8a63-194956985664" containerName="console" Jul 12 16:28:15.980374 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.980271 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2758f0-29d1-4670-8a63-194956985664" containerName="console" Jul 12 16:28:15.984541 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.984522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:15.987487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.987466 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 12 16:28:15.987602 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.987494 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-crj8n\"" Jul 12 16:28:15.988362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.988337 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 12 16:28:15.992800 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:15.992762 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf"] Jul 12 16:28:16.085856 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.085822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.085856 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.085860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rgc7\" (UniqueName: \"kubernetes.io/projected/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-kube-api-access-4rgc7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.086060 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.085881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.186826 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.186796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.186948 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.186936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.186990 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.186957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rgc7\" (UniqueName: \"kubernetes.io/projected/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-kube-api-access-4rgc7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.187222 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.187178 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.187313 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.187281 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.194857 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.194840 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rgc7\" (UniqueName: \"kubernetes.io/projected/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-kube-api-access-4rgc7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.294634 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.294552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:16.418986 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.418963 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf"] Jul 12 16:28:16.421434 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:28:16.421409 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5740c6_aaad_4c8c_a466_bab0a70c3b6c.slice/crio-074b2221eda64b296748a7bbb2801fdc871034d8aaae97dad390fb0d142e2ff4 WatchSource:0}: Error finding container 074b2221eda64b296748a7bbb2801fdc871034d8aaae97dad390fb0d142e2ff4: Status 404 returned error can't find the container with id 074b2221eda64b296748a7bbb2801fdc871034d8aaae97dad390fb0d142e2ff4 Jul 12 16:28:16.756669 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:16.756636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" event={"ID":"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c","Type":"ContainerStarted","Data":"074b2221eda64b296748a7bbb2801fdc871034d8aaae97dad390fb0d142e2ff4"} Jul 12 16:28:22.782347 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:22.782312 2575 generic.go:356] "Generic (PLEG): container finished" podID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerID="85038ab5bdca9458e1b7815cd80d8aa8708313e02770a537954c3c5fb068b513" exitCode=0 Jul 12 16:28:22.782808 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:22.782366 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" event={"ID":"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c","Type":"ContainerDied","Data":"85038ab5bdca9458e1b7815cd80d8aa8708313e02770a537954c3c5fb068b513"} Jul 12 16:28:24.792069 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:24.791976 2575 generic.go:356] "Generic (PLEG): container finished" podID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerID="4adc2989db9ebb4423b42bba48d3af612d64043012ae6345687be3cc84fec1f4" exitCode=0 Jul 12 16:28:24.792069 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:24.792025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" event={"ID":"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c","Type":"ContainerDied","Data":"4adc2989db9ebb4423b42bba48d3af612d64043012ae6345687be3cc84fec1f4"} Jul 12 16:28:26.711505 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:26.711472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:31.823020 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:31.822989 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" event={"ID":"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c","Type":"ContainerStarted","Data":"386a2082202c0e552d38d2b7869ae5b41f1f9c7a713df4435af129b1eaac3a9a"} Jul 12 16:28:31.839828 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:31.839771 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" podStartSLOduration=2.373187927 podStartE2EDuration="16.839748913s" totalImagesPullingTime="14.466560986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:28:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:28:31.839430679 +0000 UTC m=+210.608858574" watchObservedRunningTime="2026-07-12 16:28:31.839748913 +0000 UTC m=+210.609176810" Jul 12 16:28:32.828472 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:32.828438 2575 generic.go:356] "Generic (PLEG): container finished" podID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerID="386a2082202c0e552d38d2b7869ae5b41f1f9c7a713df4435af129b1eaac3a9a" exitCode=0 Jul 12 16:28:32.828912 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:32.828484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" event={"ID":"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c","Type":"ContainerDied","Data":"386a2082202c0e552d38d2b7869ae5b41f1f9c7a713df4435af129b1eaac3a9a"} Jul 12 16:28:33.950785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:33.950759 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:34.029802 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.029768 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rgc7\" (UniqueName: \"kubernetes.io/projected/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-kube-api-access-4rgc7\") pod \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " Jul 12 16:28:34.029802 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.029807 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-util\") pod \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " Jul 12 16:28:34.030049 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.029860 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-bundle\") pod \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\" (UID: \"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c\") " Jul 12 16:28:34.030657 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.030632 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-bundle" (OuterVolumeSpecName: "bundle") pod "0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" (UID: "0b5740c6-aaad-4c8c-a466-bab0a70c3b6c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:28:34.032009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.031981 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-kube-api-access-4rgc7" (OuterVolumeSpecName: "kube-api-access-4rgc7") pod "0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" (UID: "0b5740c6-aaad-4c8c-a466-bab0a70c3b6c"). InnerVolumeSpecName "kube-api-access-4rgc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:28:34.034570 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.034544 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-util" (OuterVolumeSpecName: "util") pod "0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" (UID: "0b5740c6-aaad-4c8c-a466-bab0a70c3b6c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:28:34.130702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.130622 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rgc7\" (UniqueName: \"kubernetes.io/projected/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-kube-api-access-4rgc7\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:28:34.130702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.130653 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-util\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:28:34.130702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.130663 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b5740c6-aaad-4c8c-a466-bab0a70c3b6c-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:28:34.836257 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.836231 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" Jul 12 16:28:34.836417 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.836224 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tzgcf" event={"ID":"0b5740c6-aaad-4c8c-a466-bab0a70c3b6c","Type":"ContainerDied","Data":"074b2221eda64b296748a7bbb2801fdc871034d8aaae97dad390fb0d142e2ff4"} Jul 12 16:28:34.836417 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:34.836346 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="074b2221eda64b296748a7bbb2801fdc871034d8aaae97dad390fb0d142e2ff4" Jul 12 16:28:36.970532 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:36.970500 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:37.802990 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:37.802955 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:38.893956 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.893923 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj"] Jul 12 16:28:38.894332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.894226 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerName="util" Jul 12 16:28:38.894332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.894240 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerName="util" Jul 12 16:28:38.894332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.894258 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerName="pull" Jul 12 16:28:38.894332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.894266 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerName="pull" Jul 12 16:28:38.894332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.894276 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerName="extract" Jul 12 16:28:38.894332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.894282 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerName="extract" Jul 12 16:28:38.894332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.894327 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5740c6-aaad-4c8c-a466-bab0a70c3b6c" containerName="extract" Jul 12 16:28:38.937587 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.937549 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj"] Jul 12 16:28:38.937713 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.937643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" Jul 12 16:28:38.940781 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.940747 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hf4pn\"" Jul 12 16:28:38.940781 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.940774 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 12 16:28:38.940976 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.940747 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 12 16:28:38.967325 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.967296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlsm7\" (UniqueName: \"kubernetes.io/projected/96822322-5f87-4483-a207-cf5f4b0d7a17-kube-api-access-zlsm7\") pod \"cert-manager-operator-controller-manager-55c444bf86-n82vj\" (UID: \"96822322-5f87-4483-a207-cf5f4b0d7a17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" Jul 12 16:28:38.967460 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:38.967344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96822322-5f87-4483-a207-cf5f4b0d7a17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n82vj\" (UID: \"96822322-5f87-4483-a207-cf5f4b0d7a17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" Jul 12 16:28:39.068478 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:39.068439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlsm7\" (UniqueName: \"kubernetes.io/projected/96822322-5f87-4483-a207-cf5f4b0d7a17-kube-api-access-zlsm7\") pod \"cert-manager-operator-controller-manager-55c444bf86-n82vj\" (UID: \"96822322-5f87-4483-a207-cf5f4b0d7a17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" Jul 12 16:28:39.068638 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:39.068486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96822322-5f87-4483-a207-cf5f4b0d7a17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n82vj\" (UID: \"96822322-5f87-4483-a207-cf5f4b0d7a17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" Jul 12 16:28:39.068846 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:39.068829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96822322-5f87-4483-a207-cf5f4b0d7a17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n82vj\" (UID: \"96822322-5f87-4483-a207-cf5f4b0d7a17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" Jul 12 16:28:39.079059 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:39.079020 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlsm7\" (UniqueName: \"kubernetes.io/projected/96822322-5f87-4483-a207-cf5f4b0d7a17-kube-api-access-zlsm7\") pod \"cert-manager-operator-controller-manager-55c444bf86-n82vj\" (UID: \"96822322-5f87-4483-a207-cf5f4b0d7a17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" Jul 12 16:28:39.246572 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:39.246531 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" Jul 12 16:28:39.397174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:39.397140 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj"] Jul 12 16:28:39.401268 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:28:39.401241 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96822322_5f87_4483_a207_cf5f4b0d7a17.slice/crio-0ddc4b004206829a8bfd13d882ac11357431839d1f313cc02b4aac9e82644995 WatchSource:0}: Error finding container 0ddc4b004206829a8bfd13d882ac11357431839d1f313cc02b4aac9e82644995: Status 404 returned error can't find the container with id 0ddc4b004206829a8bfd13d882ac11357431839d1f313cc02b4aac9e82644995 Jul 12 16:28:39.855089 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:39.855054 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" event={"ID":"96822322-5f87-4483-a207-cf5f4b0d7a17","Type":"ContainerStarted","Data":"0ddc4b004206829a8bfd13d882ac11357431839d1f313cc02b4aac9e82644995"} Jul 12 16:28:40.580132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:40.580086 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:40.681606 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:40.681572 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:41.873732 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:41.873656 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" event={"ID":"96822322-5f87-4483-a207-cf5f4b0d7a17","Type":"ContainerStarted","Data":"0a4ca24a2f280a83abba7d61eab200966ae8bdbff52d52735a5a8e99c6a5af7b"} Jul 12 16:28:41.891255 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:41.891158 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n82vj" podStartSLOduration=1.845436509 podStartE2EDuration="3.891143484s" totalImagesPullingTime="2.045706975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:28:41.889768911 +0000 UTC m=+220.659196803" watchObservedRunningTime="2026-07-12 16:28:41.891143484 +0000 UTC m=+220.660571389" Jul 12 16:28:42.289592 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.289554 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:42.391208 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.391158 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:42.494861 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.494782 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:42.744174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.744146 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qtc87"] Jul 12 16:28:42.766903 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.766836 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:42.770723 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.770703 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-c5ml2\"" Jul 12 16:28:42.771418 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.771405 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 12 16:28:42.775167 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.775145 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 12 16:28:42.779163 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.779145 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qtc87"] Jul 12 16:28:42.799093 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.799070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qtc87\" (UID: \"6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:42.799217 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.799123 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5z5n\" (UniqueName: \"kubernetes.io/projected/6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3-kube-api-access-s5z5n\") pod \"cert-manager-webhook-6486ffd997-qtc87\" (UID: \"6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:42.900185 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.900152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qtc87\" (UID: \"6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:42.900617 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.900226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5z5n\" (UniqueName: \"kubernetes.io/projected/6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3-kube-api-access-s5z5n\") pod \"cert-manager-webhook-6486ffd997-qtc87\" (UID: \"6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:42.908794 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.908768 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qtc87\" (UID: \"6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:42.909046 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:42.909028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5z5n\" (UniqueName: \"kubernetes.io/projected/6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3-kube-api-access-s5z5n\") pod \"cert-manager-webhook-6486ffd997-qtc87\" (UID: \"6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:43.006446 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.006409 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7"] Jul 12 16:28:43.023875 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.023802 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7"] Jul 12 16:28:43.024029 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.023935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.026632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.026612 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 12 16:28:43.026632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.026625 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 12 16:28:43.026834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.026613 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-crj8n\"" Jul 12 16:28:43.091549 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.091516 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:43.101932 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.101902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrppl\" (UniqueName: \"kubernetes.io/projected/245f6437-0208-4c54-82df-bc3fd1466ba2-kube-api-access-zrppl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.102051 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.101942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.102051 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.101969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.203103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.203071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrppl\" (UniqueName: \"kubernetes.io/projected/245f6437-0208-4c54-82df-bc3fd1466ba2-kube-api-access-zrppl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.203266 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.203130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.203266 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.203178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.203619 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.203591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.203720 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.203636 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.213636 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.213605 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrppl\" (UniqueName: \"kubernetes.io/projected/245f6437-0208-4c54-82df-bc3fd1466ba2-kube-api-access-zrppl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.233801 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.233775 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qtc87"] Jul 12 16:28:43.237295 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:28:43.237272 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f586f97_9034_4deb_b5b5_fcb6f9ab9dc3.slice/crio-a6bf6b474984cde6545cdbb4f39347cb4542cede1ce8bf6edabbe555aed59c06 WatchSource:0}: Error finding container a6bf6b474984cde6545cdbb4f39347cb4542cede1ce8bf6edabbe555aed59c06: Status 404 returned error can't find the container with id a6bf6b474984cde6545cdbb4f39347cb4542cede1ce8bf6edabbe555aed59c06 Jul 12 16:28:43.333472 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.333379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:43.474389 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.474362 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7"] Jul 12 16:28:43.476716 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:28:43.476695 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245f6437_0208_4c54_82df_bc3fd1466ba2.slice/crio-17e156f36a09ac90a29f209c4544e5ab14d77b91e02c3028f05845f52a1bc465 WatchSource:0}: Error finding container 17e156f36a09ac90a29f209c4544e5ab14d77b91e02c3028f05845f52a1bc465: Status 404 returned error can't find the container with id 17e156f36a09ac90a29f209c4544e5ab14d77b91e02c3028f05845f52a1bc465 Jul 12 16:28:43.879261 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.879148 2575 generic.go:356] "Generic (PLEG): container finished" podID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerID="7b2f602c72528c9f620ad722541e4e06c8b7d78265cd32234fd9eb67e38ed0f1" exitCode=0 Jul 12 16:28:43.879261 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.879224 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" event={"ID":"245f6437-0208-4c54-82df-bc3fd1466ba2","Type":"ContainerDied","Data":"7b2f602c72528c9f620ad722541e4e06c8b7d78265cd32234fd9eb67e38ed0f1"} Jul 12 16:28:43.879565 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.879279 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" event={"ID":"245f6437-0208-4c54-82df-bc3fd1466ba2","Type":"ContainerStarted","Data":"17e156f36a09ac90a29f209c4544e5ab14d77b91e02c3028f05845f52a1bc465"} Jul 12 16:28:43.881021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:43.880950 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" event={"ID":"6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3","Type":"ContainerStarted","Data":"a6bf6b474984cde6545cdbb4f39347cb4542cede1ce8bf6edabbe555aed59c06"} Jul 12 16:28:44.776833 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:44.776801 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:44.987099 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:44.987063 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:46.896792 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:46.896695 2575 generic.go:356] "Generic (PLEG): container finished" podID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerID="6e22958b1280dd6561cfcfdc566a8ad01033dc9015423ada29d0eda11fe8c8e0" exitCode=0 Jul 12 16:28:46.897215 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:46.896785 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" event={"ID":"245f6437-0208-4c54-82df-bc3fd1466ba2","Type":"ContainerDied","Data":"6e22958b1280dd6561cfcfdc566a8ad01033dc9015423ada29d0eda11fe8c8e0"} Jul 12 16:28:46.898446 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:46.898421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" event={"ID":"6f586f97-9034-4deb-b5b5-fcb6f9ab9dc3","Type":"ContainerStarted","Data":"790d7f51a38fbb7554b052e9b73a2aad7ec42ca902ecf3ef5fa03922ff83cdff"} Jul 12 16:28:46.898589 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:46.898556 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:46.928725 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:46.928679 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" podStartSLOduration=1.542005702 podStartE2EDuration="4.928664812s" totalImagesPullingTime="3.38665911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:28:46.927913311 +0000 UTC m=+225.697341208" watchObservedRunningTime="2026-07-12 16:28:46.928664812 +0000 UTC m=+225.698092706" Jul 12 16:28:47.368583 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:47.368556 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:47.422052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:47.421999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:47.903901 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:47.903869 2575 generic.go:356] "Generic (PLEG): container finished" podID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerID="44521e7657db5fb778784dbd83f2298e6056dc430a94bd783877f9cbda2fdf45" exitCode=0 Jul 12 16:28:47.904323 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:47.903953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" event={"ID":"245f6437-0208-4c54-82df-bc3fd1466ba2","Type":"ContainerDied","Data":"44521e7657db5fb778784dbd83f2298e6056dc430a94bd783877f9cbda2fdf45"} Jul 12 16:28:49.033013 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.032989 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:49.155497 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.155465 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-util\") pod \"245f6437-0208-4c54-82df-bc3fd1466ba2\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " Jul 12 16:28:49.155497 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.155504 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrppl\" (UniqueName: \"kubernetes.io/projected/245f6437-0208-4c54-82df-bc3fd1466ba2-kube-api-access-zrppl\") pod \"245f6437-0208-4c54-82df-bc3fd1466ba2\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " Jul 12 16:28:49.155744 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.155537 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-bundle\") pod \"245f6437-0208-4c54-82df-bc3fd1466ba2\" (UID: \"245f6437-0208-4c54-82df-bc3fd1466ba2\") " Jul 12 16:28:49.155951 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.155923 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-bundle" (OuterVolumeSpecName: "bundle") pod "245f6437-0208-4c54-82df-bc3fd1466ba2" (UID: "245f6437-0208-4c54-82df-bc3fd1466ba2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:28:49.157648 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.157618 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245f6437-0208-4c54-82df-bc3fd1466ba2-kube-api-access-zrppl" (OuterVolumeSpecName: "kube-api-access-zrppl") pod "245f6437-0208-4c54-82df-bc3fd1466ba2" (UID: "245f6437-0208-4c54-82df-bc3fd1466ba2"). InnerVolumeSpecName "kube-api-access-zrppl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:28:49.160342 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.160276 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-util" (OuterVolumeSpecName: "util") pod "245f6437-0208-4c54-82df-bc3fd1466ba2" (UID: "245f6437-0208-4c54-82df-bc3fd1466ba2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:28:49.256341 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.256297 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-util\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:28:49.256341 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.256332 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrppl\" (UniqueName: \"kubernetes.io/projected/245f6437-0208-4c54-82df-bc3fd1466ba2-kube-api-access-zrppl\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:28:49.256341 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.256348 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245f6437-0208-4c54-82df-bc3fd1466ba2-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:28:49.555132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.555094 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d"] Jul 12 16:28:49.555460 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.555442 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerName="extract" Jul 12 16:28:49.555539 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.555461 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerName="extract" Jul 12 16:28:49.555539 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.555485 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerName="util" Jul 12 16:28:49.555539 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.555493 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerName="util" Jul 12 16:28:49.555539 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.555501 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerName="pull" Jul 12 16:28:49.555539 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.555509 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerName="pull" Jul 12 16:28:49.555771 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.555578 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="245f6437-0208-4c54-82df-bc3fd1466ba2" containerName="extract" Jul 12 16:28:49.557386 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.557364 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" Jul 12 16:28:49.559943 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.559926 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7xt77\"" Jul 12 16:28:49.566537 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.566514 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d"] Jul 12 16:28:49.659268 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.659228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92ce3cb2-c477-416f-85f0-3e9ae8957bb7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-s9x4d\" (UID: \"92ce3cb2-c477-416f-85f0-3e9ae8957bb7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" Jul 12 16:28:49.659435 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.659279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb6kw\" (UniqueName: \"kubernetes.io/projected/92ce3cb2-c477-416f-85f0-3e9ae8957bb7-kube-api-access-bb6kw\") pod \"cert-manager-cainjector-559cdd8c87-s9x4d\" (UID: \"92ce3cb2-c477-416f-85f0-3e9ae8957bb7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" Jul 12 16:28:49.760433 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.760397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92ce3cb2-c477-416f-85f0-3e9ae8957bb7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-s9x4d\" (UID: \"92ce3cb2-c477-416f-85f0-3e9ae8957bb7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" Jul 12 16:28:49.760597 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.760454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6kw\" (UniqueName: \"kubernetes.io/projected/92ce3cb2-c477-416f-85f0-3e9ae8957bb7-kube-api-access-bb6kw\") pod \"cert-manager-cainjector-559cdd8c87-s9x4d\" (UID: \"92ce3cb2-c477-416f-85f0-3e9ae8957bb7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" Jul 12 16:28:49.775035 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.774998 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92ce3cb2-c477-416f-85f0-3e9ae8957bb7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-s9x4d\" (UID: \"92ce3cb2-c477-416f-85f0-3e9ae8957bb7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" Jul 12 16:28:49.775142 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.775079 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6kw\" (UniqueName: \"kubernetes.io/projected/92ce3cb2-c477-416f-85f0-3e9ae8957bb7-kube-api-access-bb6kw\") pod \"cert-manager-cainjector-559cdd8c87-s9x4d\" (UID: \"92ce3cb2-c477-416f-85f0-3e9ae8957bb7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" Jul 12 16:28:49.866830 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.866757 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" Jul 12 16:28:49.915938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.915913 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" event={"ID":"245f6437-0208-4c54-82df-bc3fd1466ba2","Type":"ContainerDied","Data":"17e156f36a09ac90a29f209c4544e5ab14d77b91e02c3028f05845f52a1bc465"} Jul 12 16:28:49.916052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.915965 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17e156f36a09ac90a29f209c4544e5ab14d77b91e02c3028f05845f52a1bc465" Jul 12 16:28:49.916052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.915973 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6v4r7" Jul 12 16:28:49.998288 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:49.998257 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d"] Jul 12 16:28:50.795125 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:50.795087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:50.922277 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:50.922231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" event={"ID":"92ce3cb2-c477-416f-85f0-3e9ae8957bb7","Type":"ContainerStarted","Data":"f12f757c0fc6716aaa6523692c5908161516f6bafae64b471283ef87580d47c5"} Jul 12 16:28:50.922277 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:50.922286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" event={"ID":"92ce3cb2-c477-416f-85f0-3e9ae8957bb7","Type":"ContainerStarted","Data":"ff7cbaa9cb19fb483079e081e2feff99358f30f74d4911bdda48593035fb3090"} Jul 12 16:28:50.949785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:50.949713 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-s9x4d" podStartSLOduration=1.949693088 podStartE2EDuration="1.949693088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:28:50.947930034 +0000 UTC m=+229.717357940" watchObservedRunningTime="2026-07-12 16:28:50.949693088 +0000 UTC m=+229.719120986" Jul 12 16:28:50.950601 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:50.950563 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:52.000905 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:52.000865 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:52.906015 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:52.905987 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qtc87" Jul 12 16:28:53.060156 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.060124 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pp6rg"] Jul 12 16:28:53.063171 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.063155 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pp6rg" Jul 12 16:28:53.065611 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.065592 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8c676\"" Jul 12 16:28:53.071412 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.071390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pp6rg"] Jul 12 16:28:53.188205 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.188157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcd1f50-5335-49a0-b9c3-31c0d83740f3-bound-sa-token\") pod \"cert-manager-55b4d954d5-pp6rg\" (UID: \"adcd1f50-5335-49a0-b9c3-31c0d83740f3\") " pod="cert-manager/cert-manager-55b4d954d5-pp6rg" Jul 12 16:28:53.188353 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.188211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vphb\" (UniqueName: \"kubernetes.io/projected/adcd1f50-5335-49a0-b9c3-31c0d83740f3-kube-api-access-2vphb\") pod \"cert-manager-55b4d954d5-pp6rg\" (UID: \"adcd1f50-5335-49a0-b9c3-31c0d83740f3\") " pod="cert-manager/cert-manager-55b4d954d5-pp6rg" Jul 12 16:28:53.288666 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.288634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcd1f50-5335-49a0-b9c3-31c0d83740f3-bound-sa-token\") pod \"cert-manager-55b4d954d5-pp6rg\" (UID: \"adcd1f50-5335-49a0-b9c3-31c0d83740f3\") " pod="cert-manager/cert-manager-55b4d954d5-pp6rg" Jul 12 16:28:53.288666 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.288670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vphb\" (UniqueName: \"kubernetes.io/projected/adcd1f50-5335-49a0-b9c3-31c0d83740f3-kube-api-access-2vphb\") pod \"cert-manager-55b4d954d5-pp6rg\" (UID: \"adcd1f50-5335-49a0-b9c3-31c0d83740f3\") " pod="cert-manager/cert-manager-55b4d954d5-pp6rg" Jul 12 16:28:53.297084 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.297052 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adcd1f50-5335-49a0-b9c3-31c0d83740f3-bound-sa-token\") pod \"cert-manager-55b4d954d5-pp6rg\" (UID: \"adcd1f50-5335-49a0-b9c3-31c0d83740f3\") " pod="cert-manager/cert-manager-55b4d954d5-pp6rg" Jul 12 16:28:53.297214 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.297168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vphb\" (UniqueName: \"kubernetes.io/projected/adcd1f50-5335-49a0-b9c3-31c0d83740f3-kube-api-access-2vphb\") pod \"cert-manager-55b4d954d5-pp6rg\" (UID: \"adcd1f50-5335-49a0-b9c3-31c0d83740f3\") " pod="cert-manager/cert-manager-55b4d954d5-pp6rg" Jul 12 16:28:53.372854 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.372818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pp6rg" Jul 12 16:28:53.522033 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.521998 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pp6rg"] Jul 12 16:28:53.524885 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:28:53.524856 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadcd1f50_5335_49a0_b9c3_31c0d83740f3.slice/crio-912e157fe50e4b71de89cc14f459a8bce7d4c6f16c533686120f7a079ccb7b71 WatchSource:0}: Error finding container 912e157fe50e4b71de89cc14f459a8bce7d4c6f16c533686120f7a079ccb7b71: Status 404 returned error can't find the container with id 912e157fe50e4b71de89cc14f459a8bce7d4c6f16c533686120f7a079ccb7b71 Jul 12 16:28:53.935814 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.935782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pp6rg" event={"ID":"adcd1f50-5335-49a0-b9c3-31c0d83740f3","Type":"ContainerStarted","Data":"c7a891eeb990d2d26a12f1044e5bfc47ddb3f9a0ab0b3fb5c6348f19e0ceb9d3"} Jul 12 16:28:53.935966 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.935824 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pp6rg" event={"ID":"adcd1f50-5335-49a0-b9c3-31c0d83740f3","Type":"ContainerStarted","Data":"912e157fe50e4b71de89cc14f459a8bce7d4c6f16c533686120f7a079ccb7b71"} Jul 12 16:28:53.952761 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:53.952713 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pp6rg" podStartSLOduration=0.952699009 podStartE2EDuration="952.699009ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:28:53.952065546 +0000 UTC m=+232.721493454" watchObservedRunningTime="2026-07-12 16:28:53.952699009 +0000 UTC m=+232.722126903" Jul 12 16:28:55.119720 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:55.119681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:55.220887 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:55.220859 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:28:56.950153 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:56.950123 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2"] Jul 12 16:28:56.953114 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:56.953097 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" Jul 12 16:28:56.956666 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:56.956639 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-r9hbq\"" Jul 12 16:28:56.956797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:56.956642 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 12 16:28:56.956797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:56.956677 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 12 16:28:56.961574 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:56.961554 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2"] Jul 12 16:28:57.016870 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.016841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7pc\" (UniqueName: \"kubernetes.io/projected/5d3d199d-4eef-4182-b4c0-0c256ff3c324-kube-api-access-2k7pc\") pod \"openshift-lws-operator-fd8ccff4c-c49f2\" (UID: \"5d3d199d-4eef-4182-b4c0-0c256ff3c324\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" Jul 12 16:28:57.017020 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.016892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3d199d-4eef-4182-b4c0-0c256ff3c324-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c49f2\" (UID: \"5d3d199d-4eef-4182-b4c0-0c256ff3c324\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" Jul 12 16:28:57.118353 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.118320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7pc\" (UniqueName: \"kubernetes.io/projected/5d3d199d-4eef-4182-b4c0-0c256ff3c324-kube-api-access-2k7pc\") pod \"openshift-lws-operator-fd8ccff4c-c49f2\" (UID: \"5d3d199d-4eef-4182-b4c0-0c256ff3c324\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" Jul 12 16:28:57.118505 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.118378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3d199d-4eef-4182-b4c0-0c256ff3c324-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c49f2\" (UID: \"5d3d199d-4eef-4182-b4c0-0c256ff3c324\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" Jul 12 16:28:57.118771 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.118752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3d199d-4eef-4182-b4c0-0c256ff3c324-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c49f2\" (UID: \"5d3d199d-4eef-4182-b4c0-0c256ff3c324\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" Jul 12 16:28:57.126570 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.126538 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7pc\" (UniqueName: \"kubernetes.io/projected/5d3d199d-4eef-4182-b4c0-0c256ff3c324-kube-api-access-2k7pc\") pod \"openshift-lws-operator-fd8ccff4c-c49f2\" (UID: \"5d3d199d-4eef-4182-b4c0-0c256ff3c324\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" Jul 12 16:28:57.262727 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.262652 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" Jul 12 16:28:57.388790 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.388761 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2"] Jul 12 16:28:57.391429 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:28:57.391404 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d3d199d_4eef_4182_b4c0_0c256ff3c324.slice/crio-2cf01de2ef231491b5c5e56d284c02c71c8da89f80ee1ba2fd1a4e3a3d49e7eb WatchSource:0}: Error finding container 2cf01de2ef231491b5c5e56d284c02c71c8da89f80ee1ba2fd1a4e3a3d49e7eb: Status 404 returned error can't find the container with id 2cf01de2ef231491b5c5e56d284c02c71c8da89f80ee1ba2fd1a4e3a3d49e7eb Jul 12 16:28:57.952938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:28:57.952901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" event={"ID":"5d3d199d-4eef-4182-b4c0-0c256ff3c324","Type":"ContainerStarted","Data":"2cf01de2ef231491b5c5e56d284c02c71c8da89f80ee1ba2fd1a4e3a3d49e7eb"} Jul 12 16:29:00.967926 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:00.967888 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" event={"ID":"5d3d199d-4eef-4182-b4c0-0c256ff3c324","Type":"ContainerStarted","Data":"3a2331f965628c450351ae2f579fbe29ba16d336508a6ee15cc42c36cb858fe1"} Jul 12 16:29:00.985365 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:00.985310 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c49f2" podStartSLOduration=2.203531915 podStartE2EDuration="4.985293477s" totalImagesPullingTime="2.781761562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:29:00.984283052 +0000 UTC m=+239.753710971" watchObservedRunningTime="2026-07-12 16:29:00.985293477 +0000 UTC m=+239.754721372" Jul 12 16:29:01.608410 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:01.608370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:01.712262 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:01.712227 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:03.140009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.139959 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb"] Jul 12 16:29:03.143421 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.143405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.145763 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.145744 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 12 16:29:03.146713 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.146700 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-crj8n\"" Jul 12 16:29:03.146713 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.146708 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 12 16:29:03.151801 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.151769 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb"] Jul 12 16:29:03.268282 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.268230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.268282 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.268285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/b082a883-1686-4aea-844c-817c5ad85400-kube-api-access-plwx5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.268498 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.268367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.314859 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.314799 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:03.368962 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.368925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.369144 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.368978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.369234 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.369012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/b082a883-1686-4aea-844c-817c5ad85400-kube-api-access-plwx5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.369684 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.369657 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.369854 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.369692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.377494 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.377469 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/b082a883-1686-4aea-844c-817c5ad85400-kube-api-access-plwx5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.453448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.453413 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:03.579127 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.579100 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb"] Jul 12 16:29:03.581404 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:03.581374 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb082a883_1686_4aea_844c_817c5ad85400.slice/crio-3db0874dbebaf028fb4fc5cdaaacc918d1f77e4c7bd4783382d777953f20326e WatchSource:0}: Error finding container 3db0874dbebaf028fb4fc5cdaaacc918d1f77e4c7bd4783382d777953f20326e: Status 404 returned error can't find the container with id 3db0874dbebaf028fb4fc5cdaaacc918d1f77e4c7bd4783382d777953f20326e Jul 12 16:29:03.982119 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.982086 2575 generic.go:356] "Generic (PLEG): container finished" podID="b082a883-1686-4aea-844c-817c5ad85400" containerID="b1e39f9ecd377b243d9eeebf71f57c253dc8de47c26a8aa228cdf71680a8b98a" exitCode=0 Jul 12 16:29:03.982278 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.982137 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" event={"ID":"b082a883-1686-4aea-844c-817c5ad85400","Type":"ContainerDied","Data":"b1e39f9ecd377b243d9eeebf71f57c253dc8de47c26a8aa228cdf71680a8b98a"} Jul 12 16:29:03.982278 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:03.982204 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" event={"ID":"b082a883-1686-4aea-844c-817c5ad85400","Type":"ContainerStarted","Data":"3db0874dbebaf028fb4fc5cdaaacc918d1f77e4c7bd4783382d777953f20326e"} Jul 12 16:29:04.498301 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:04.498269 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:04.987780 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:04.987746 2575 generic.go:356] "Generic (PLEG): container finished" podID="b082a883-1686-4aea-844c-817c5ad85400" containerID="9867b845c9091d3737f80169b8d99df7d40bc615be63ee3410d3c1ab41ed7537" exitCode=0 Jul 12 16:29:04.987949 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:04.987792 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" event={"ID":"b082a883-1686-4aea-844c-817c5ad85400","Type":"ContainerDied","Data":"9867b845c9091d3737f80169b8d99df7d40bc615be63ee3410d3c1ab41ed7537"} Jul 12 16:29:05.994428 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:05.994393 2575 generic.go:356] "Generic (PLEG): container finished" podID="b082a883-1686-4aea-844c-817c5ad85400" containerID="f6ee5fe78390883e2f35a5e7a4a63dd3c97637bbf14396e10d3da6b5be00e48f" exitCode=0 Jul 12 16:29:05.994859 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:05.994477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" event={"ID":"b082a883-1686-4aea-844c-817c5ad85400","Type":"ContainerDied","Data":"f6ee5fe78390883e2f35a5e7a4a63dd3c97637bbf14396e10d3da6b5be00e48f"} Jul 12 16:29:07.125155 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.125131 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:07.202424 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.202394 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/b082a883-1686-4aea-844c-817c5ad85400-kube-api-access-plwx5\") pod \"b082a883-1686-4aea-844c-817c5ad85400\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " Jul 12 16:29:07.202586 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.202438 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-bundle\") pod \"b082a883-1686-4aea-844c-817c5ad85400\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " Jul 12 16:29:07.202586 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.202470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-util\") pod \"b082a883-1686-4aea-844c-817c5ad85400\" (UID: \"b082a883-1686-4aea-844c-817c5ad85400\") " Jul 12 16:29:07.203120 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.203090 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-bundle" (OuterVolumeSpecName: "bundle") pod "b082a883-1686-4aea-844c-817c5ad85400" (UID: "b082a883-1686-4aea-844c-817c5ad85400"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:29:07.204503 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.204482 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b082a883-1686-4aea-844c-817c5ad85400-kube-api-access-plwx5" (OuterVolumeSpecName: "kube-api-access-plwx5") pod "b082a883-1686-4aea-844c-817c5ad85400" (UID: "b082a883-1686-4aea-844c-817c5ad85400"). InnerVolumeSpecName "kube-api-access-plwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:29:07.207990 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.207959 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-util" (OuterVolumeSpecName: "util") pod "b082a883-1686-4aea-844c-817c5ad85400" (UID: "b082a883-1686-4aea-844c-817c5ad85400"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:29:07.304023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.303942 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwx5\" (UniqueName: \"kubernetes.io/projected/b082a883-1686-4aea-844c-817c5ad85400-kube-api-access-plwx5\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:29:07.304023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.303970 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:29:07.304023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:07.303979 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b082a883-1686-4aea-844c-817c5ad85400-util\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:29:08.004402 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:08.004367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" event={"ID":"b082a883-1686-4aea-844c-817c5ad85400","Type":"ContainerDied","Data":"3db0874dbebaf028fb4fc5cdaaacc918d1f77e4c7bd4783382d777953f20326e"} Jul 12 16:29:08.004402 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:08.004395 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af494mb" Jul 12 16:29:08.004650 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:08.004421 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db0874dbebaf028fb4fc5cdaaacc918d1f77e4c7bd4783382d777953f20326e" Jul 12 16:29:16.490733 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:16.490700 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:16.593168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:16.593133 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:17.089743 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.089710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:17.373309 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.373208 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:17.475972 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.475936 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:17.501083 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.501053 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv"] Jul 12 16:29:17.501464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.501348 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b082a883-1686-4aea-844c-817c5ad85400" containerName="pull" Jul 12 16:29:17.501464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.501362 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b082a883-1686-4aea-844c-817c5ad85400" containerName="pull" Jul 12 16:29:17.501464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.501374 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b082a883-1686-4aea-844c-817c5ad85400" containerName="util" Jul 12 16:29:17.501464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.501380 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b082a883-1686-4aea-844c-817c5ad85400" containerName="util" Jul 12 16:29:17.501464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.501397 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b082a883-1686-4aea-844c-817c5ad85400" containerName="extract" Jul 12 16:29:17.501464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.501403 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b082a883-1686-4aea-844c-817c5ad85400" containerName="extract" Jul 12 16:29:17.501464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.501448 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b082a883-1686-4aea-844c-817c5ad85400" containerName="extract" Jul 12 16:29:17.547245 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.547215 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv"] Jul 12 16:29:17.547383 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.547335 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.550179 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.550153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xbsdr\"" Jul 12 16:29:17.550341 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.550200 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 12 16:29:17.550450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.550434 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 12 16:29:17.550491 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.550465 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 12 16:29:17.550672 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.550660 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 12 16:29:17.662566 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.662485 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:17.693579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.693551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/142a09a0-7896-4aed-bfe0-6001e4ad09d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.693665 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.693591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llvnf\" (UniqueName: \"kubernetes.io/projected/142a09a0-7896-4aed-bfe0-6001e4ad09d3-kube-api-access-llvnf\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.693707 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.693651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/142a09a0-7896-4aed-bfe0-6001e4ad09d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.693753 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.693737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/142a09a0-7896-4aed-bfe0-6001e4ad09d3-manifests\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.693794 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.693784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/142a09a0-7896-4aed-bfe0-6001e4ad09d3-tmp\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.762462 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.762425 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:17.794376 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.794345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvnf\" (UniqueName: \"kubernetes.io/projected/142a09a0-7896-4aed-bfe0-6001e4ad09d3-kube-api-access-llvnf\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.794492 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.794382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/142a09a0-7896-4aed-bfe0-6001e4ad09d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.794492 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.794429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/142a09a0-7896-4aed-bfe0-6001e4ad09d3-manifests\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.794492 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.794455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/142a09a0-7896-4aed-bfe0-6001e4ad09d3-tmp\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.794492 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.794485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/142a09a0-7896-4aed-bfe0-6001e4ad09d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.794883 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.794857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/142a09a0-7896-4aed-bfe0-6001e4ad09d3-manifests\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.796833 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.796808 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/142a09a0-7896-4aed-bfe0-6001e4ad09d3-tmp\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.796985 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.796964 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/142a09a0-7896-4aed-bfe0-6001e4ad09d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.797049 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.797033 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/142a09a0-7896-4aed-bfe0-6001e4ad09d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.802669 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.802649 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvnf\" (UniqueName: \"kubernetes.io/projected/142a09a0-7896-4aed-bfe0-6001e4ad09d3-kube-api-access-llvnf\") pod \"opendatahub-operator-controller-manager-869b4c4546-2x5pv\" (UID: \"142a09a0-7896-4aed-bfe0-6001e4ad09d3\") " pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.857639 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.857604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:17.863269 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.863240 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:17.983294 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:17.983183 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:18.056312 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:18.056288 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv"] Jul 12 16:29:18.058614 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:18.058583 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142a09a0_7896_4aed_bfe0_6001e4ad09d3.slice/crio-18377305d27ee04f6aa1d0383c36cc698f26dc6a86e8d253f0d96226f58e65ba WatchSource:0}: Error finding container 18377305d27ee04f6aa1d0383c36cc698f26dc6a86e8d253f0d96226f58e65ba: Status 404 returned error can't find the container with id 18377305d27ee04f6aa1d0383c36cc698f26dc6a86e8d253f0d96226f58e65ba Jul 12 16:29:18.161004 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:18.160972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:18.265217 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:18.265121 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:19.053672 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:19.053632 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" event={"ID":"142a09a0-7896-4aed-bfe0-6001e4ad09d3","Type":"ContainerStarted","Data":"18377305d27ee04f6aa1d0383c36cc698f26dc6a86e8d253f0d96226f58e65ba"} Jul 12 16:29:21.063705 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:21.063672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" event={"ID":"142a09a0-7896-4aed-bfe0-6001e4ad09d3","Type":"ContainerStarted","Data":"9dd4ad3476cf82169889cda6236fc1b2411c408182f34b8830131bb8ad295690"} Jul 12 16:29:23.073543 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:23.073508 2575 generic.go:356] "Generic (PLEG): container finished" podID="142a09a0-7896-4aed-bfe0-6001e4ad09d3" containerID="9dd4ad3476cf82169889cda6236fc1b2411c408182f34b8830131bb8ad295690" exitCode=0 Jul 12 16:29:23.073936 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:23.073579 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" event={"ID":"142a09a0-7896-4aed-bfe0-6001e4ad09d3","Type":"ContainerDied","Data":"9dd4ad3476cf82169889cda6236fc1b2411c408182f34b8830131bb8ad295690"} Jul 12 16:29:23.663571 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:23.663540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:23.829590 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:23.829552 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:24.079515 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:24.079480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" event={"ID":"142a09a0-7896-4aed-bfe0-6001e4ad09d3","Type":"ContainerStarted","Data":"d49ec4a998143ca32c81fe7ac62568a0a645615d24ebbb605853037f8d5df0fa"} Jul 12 16:29:24.079918 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:24.079692 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:24.117477 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:24.117428 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" podStartSLOduration=2.789155285 podStartE2EDuration="7.117411917s" totalImagesPullingTime="3.23809247s" totalInitContainerRuntime="1.090164162s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:29:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:29:24.115032362 +0000 UTC m=+262.884460256" watchObservedRunningTime="2026-07-12 16:29:24.117411917 +0000 UTC m=+262.886839844" Jul 12 16:29:26.422319 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:26.422275 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:27.581205 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:27.581172 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:29.239630 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:29.239596 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:29.348652 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:29.348614 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:35.086799 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:35.086770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-869b4c4546-2x5pv" Jul 12 16:29:36.672581 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:36.672530 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:36.780074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:36.780038 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:37.298009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.297973 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wccbh"] Jul 12 16:29:37.305599 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.305574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:37.308307 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.308276 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 12 16:29:37.308531 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.308515 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tn6v7\"" Jul 12 16:29:37.321299 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.321271 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wccbh"] Jul 12 16:29:37.343780 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.343751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00b243a7-1af7-4dbf-9ca3-5a69756185fa-cert\") pod \"odh-model-controller-7998c67dc4-wccbh\" (UID: \"00b243a7-1af7-4dbf-9ca3-5a69756185fa\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:37.343915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.343820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcv7t\" (UniqueName: \"kubernetes.io/projected/00b243a7-1af7-4dbf-9ca3-5a69756185fa-kube-api-access-dcv7t\") pod \"odh-model-controller-7998c67dc4-wccbh\" (UID: \"00b243a7-1af7-4dbf-9ca3-5a69756185fa\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:37.370525 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.370492 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:37.444804 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.444757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcv7t\" (UniqueName: \"kubernetes.io/projected/00b243a7-1af7-4dbf-9ca3-5a69756185fa-kube-api-access-dcv7t\") pod \"odh-model-controller-7998c67dc4-wccbh\" (UID: \"00b243a7-1af7-4dbf-9ca3-5a69756185fa\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:37.444804 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.444816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00b243a7-1af7-4dbf-9ca3-5a69756185fa-cert\") pod \"odh-model-controller-7998c67dc4-wccbh\" (UID: \"00b243a7-1af7-4dbf-9ca3-5a69756185fa\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:37.445012 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:29:37.444935 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 12 16:29:37.445012 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:29:37.444998 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b243a7-1af7-4dbf-9ca3-5a69756185fa-cert podName:00b243a7-1af7-4dbf-9ca3-5a69756185fa nodeName:}" failed. No retries permitted until 2026-07-12 16:29:37.944980085 +0000 UTC m=+276.714407957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/00b243a7-1af7-4dbf-9ca3-5a69756185fa-cert") pod "odh-model-controller-7998c67dc4-wccbh" (UID: "00b243a7-1af7-4dbf-9ca3-5a69756185fa") : secret "odh-model-controller-webhook-cert" not found Jul 12 16:29:37.452772 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.452744 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:37.454005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.453984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcv7t\" (UniqueName: \"kubernetes.io/projected/00b243a7-1af7-4dbf-9ca3-5a69756185fa-kube-api-access-dcv7t\") pod \"odh-model-controller-7998c67dc4-wccbh\" (UID: \"00b243a7-1af7-4dbf-9ca3-5a69756185fa\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:37.624351 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.624240 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:37.801482 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.801446 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:37.860023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.859986 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x926v"] Jul 12 16:29:37.863199 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.863171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:37.865911 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.865889 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pg7nn\"" Jul 12 16:29:37.866039 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.865916 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 12 16:29:37.874798 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.874742 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x926v"] Jul 12 16:29:37.904397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.904363 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:37.947579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.947541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lv2\" (UniqueName: \"kubernetes.io/projected/e9ed3e70-852f-4188-8568-0e724d84eef6-kube-api-access-v2lv2\") pod \"model-serving-api-54d89bd5fc-x926v\" (UID: \"e9ed3e70-852f-4188-8568-0e724d84eef6\") " pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:37.947728 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.947599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ed3e70-852f-4188-8568-0e724d84eef6-tls-certs\") pod \"model-serving-api-54d89bd5fc-x926v\" (UID: \"e9ed3e70-852f-4188-8568-0e724d84eef6\") " pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:37.947728 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.947662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00b243a7-1af7-4dbf-9ca3-5a69756185fa-cert\") pod \"odh-model-controller-7998c67dc4-wccbh\" (UID: \"00b243a7-1af7-4dbf-9ca3-5a69756185fa\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:37.950097 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:37.950072 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00b243a7-1af7-4dbf-9ca3-5a69756185fa-cert\") pod \"odh-model-controller-7998c67dc4-wccbh\" (UID: \"00b243a7-1af7-4dbf-9ca3-5a69756185fa\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:38.048435 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:38.048394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lv2\" (UniqueName: \"kubernetes.io/projected/e9ed3e70-852f-4188-8568-0e724d84eef6-kube-api-access-v2lv2\") pod \"model-serving-api-54d89bd5fc-x926v\" (UID: \"e9ed3e70-852f-4188-8568-0e724d84eef6\") " pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:38.048626 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:38.048479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ed3e70-852f-4188-8568-0e724d84eef6-tls-certs\") pod \"model-serving-api-54d89bd5fc-x926v\" (UID: \"e9ed3e70-852f-4188-8568-0e724d84eef6\") " pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:38.050919 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:38.050891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ed3e70-852f-4188-8568-0e724d84eef6-tls-certs\") pod \"model-serving-api-54d89bd5fc-x926v\" (UID: \"e9ed3e70-852f-4188-8568-0e724d84eef6\") " pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:38.056749 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:38.056731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lv2\" (UniqueName: \"kubernetes.io/projected/e9ed3e70-852f-4188-8568-0e724d84eef6-kube-api-access-v2lv2\") pod \"model-serving-api-54d89bd5fc-x926v\" (UID: \"e9ed3e70-852f-4188-8568-0e724d84eef6\") " pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:38.174482 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:38.174400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:38.216077 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:38.216039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:38.313245 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:38.313218 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x926v"] Jul 12 16:29:38.316443 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:38.316418 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ed3e70_852f_4188_8568_0e724d84eef6.slice/crio-05edd5fc95679b155649fd879481f7fe2aa12dbf0d99721346adcd5385280777 WatchSource:0}: Error finding container 05edd5fc95679b155649fd879481f7fe2aa12dbf0d99721346adcd5385280777: Status 404 returned error can't find the container with id 05edd5fc95679b155649fd879481f7fe2aa12dbf0d99721346adcd5385280777 Jul 12 16:29:38.355239 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:38.355218 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wccbh"] Jul 12 16:29:38.357044 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:38.357020 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b243a7_1af7_4dbf_9ca3_5a69756185fa.slice/crio-9927f12f1032ab27014e4ebc4219a3e06f3f5d87616b624f250787651e320c96 WatchSource:0}: Error finding container 9927f12f1032ab27014e4ebc4219a3e06f3f5d87616b624f250787651e320c96: Status 404 returned error can't find the container with id 9927f12f1032ab27014e4ebc4219a3e06f3f5d87616b624f250787651e320c96 Jul 12 16:29:39.150446 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:39.150382 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x926v" event={"ID":"e9ed3e70-852f-4188-8568-0e724d84eef6","Type":"ContainerStarted","Data":"05edd5fc95679b155649fd879481f7fe2aa12dbf0d99721346adcd5385280777"} Jul 12 16:29:39.153218 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:39.153108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" event={"ID":"00b243a7-1af7-4dbf-9ca3-5a69756185fa","Type":"ContainerStarted","Data":"9927f12f1032ab27014e4ebc4219a3e06f3f5d87616b624f250787651e320c96"} Jul 12 16:29:41.163737 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:41.163700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" event={"ID":"00b243a7-1af7-4dbf-9ca3-5a69756185fa","Type":"ContainerStarted","Data":"9f6f29867679c78715d3c0a34c470306b3fb629d191e3f166fc099ea28bd68b7"} Jul 12 16:29:41.164154 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:41.163809 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:41.165279 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:41.165256 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x926v" event={"ID":"e9ed3e70-852f-4188-8568-0e724d84eef6","Type":"ContainerStarted","Data":"c04d046de60480d43797c077c200152ca43292bffae8b7868f6f776966a64b55"} Jul 12 16:29:41.165408 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:41.165392 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:41.179098 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:41.179059 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" podStartSLOduration=1.462137536 podStartE2EDuration="4.179048426s" totalImagesPullingTime="2.71691089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:29:41.177604588 +0000 UTC m=+279.947032483" watchObservedRunningTime="2026-07-12 16:29:41.179048426 +0000 UTC m=+279.948476298" Jul 12 16:29:41.193434 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:41.193395 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-x926v" podStartSLOduration=1.48428738 podStartE2EDuration="4.193382685s" totalImagesPullingTime="2.709095305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:29:41.192563523 +0000 UTC m=+279.961991431" watchObservedRunningTime="2026-07-12 16:29:41.193382685 +0000 UTC m=+279.962810580" Jul 12 16:29:42.463072 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.463040 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:42.530478 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.530401 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-545795f48c-vwbng"] Jul 12 16:29:42.533782 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.533762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.536530 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.536511 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 12 16:29:42.536647 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.536559 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 12 16:29:42.537403 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.537386 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 12 16:29:42.537479 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.537411 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xflvk\"" Jul 12 16:29:42.537479 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.537419 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 12 16:29:42.545694 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.545676 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545795f48c-vwbng"] Jul 12 16:29:42.591962 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.591931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jm4q\" (UniqueName: \"kubernetes.io/projected/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-kube-api-access-8jm4q\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.591962 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.591964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tmp\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.592147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.592068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tls-certs\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.692487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.692448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tls-certs\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.692487 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.692491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jm4q\" (UniqueName: \"kubernetes.io/projected/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-kube-api-access-8jm4q\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.692739 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.692510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tmp\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.692739 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:29:42.692607 2575 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 12 16:29:42.692739 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:29:42.692684 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tls-certs podName:923f08c1-c7f8-4f05-b561-f38e4cfaa1e7 nodeName:}" failed. No retries permitted until 2026-07-12 16:29:43.192663414 +0000 UTC m=+281.962091301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tls-certs") pod "kube-auth-proxy-545795f48c-vwbng" (UID: "923f08c1-c7f8-4f05-b561-f38e4cfaa1e7") : secret "kube-auth-proxy-tls" not found Jul 12 16:29:42.694762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.694741 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tmp\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.701000 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.700968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jm4q\" (UniqueName: \"kubernetes.io/projected/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-kube-api-access-8jm4q\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:42.818519 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:42.818414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:43.198702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.198667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tls-certs\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:43.201102 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.201073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923f08c1-c7f8-4f05-b561-f38e4cfaa1e7-tls-certs\") pod \"kube-auth-proxy-545795f48c-vwbng\" (UID: \"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7\") " pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:43.442696 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.442657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" Jul 12 16:29:43.490162 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.490078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:43.491417 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.491050 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d"] Jul 12 16:29:43.496261 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.496235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.499457 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.498977 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 16:29:43.499457 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.499352 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9z4mj\"" Jul 12 16:29:43.499612 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.499500 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 12 16:29:43.500263 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.500213 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 12 16:29:43.500378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.500282 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 12 16:29:43.514576 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.514510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d"] Jul 12 16:29:43.578713 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.578553 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:43.601159 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.601124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.601159 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.601161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.601384 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.601222 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-token\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.601384 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.601272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-cacerts\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.601384 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.601294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kpb4\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-kube-api-access-9kpb4\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.601479 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.601404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.601479 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.601440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-local-certs\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.607339 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.607318 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545795f48c-vwbng"] Jul 12 16:29:43.609543 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:43.609517 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923f08c1_c7f8_4f05_b561_f38e4cfaa1e7.slice/crio-bccfc67a23fc6b6754a7fcfa4bab256f24825a052795f813e4566dbd4058df78 WatchSource:0}: Error finding container bccfc67a23fc6b6754a7fcfa4bab256f24825a052795f813e4566dbd4058df78: Status 404 returned error can't find the container with id bccfc67a23fc6b6754a7fcfa4bab256f24825a052795f813e4566dbd4058df78 Jul 12 16:29:43.701855 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.701821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-token\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.702009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.701864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-cacerts\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.702009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.701884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kpb4\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-kube-api-access-9kpb4\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.702009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.701940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.702009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.701959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-local-certs\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.702009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.701976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.702009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.701993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.702720 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.702691 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.704356 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.704328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.704442 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.704334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-local-certs\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.704442 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.704356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-cacerts\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.704442 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.704426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.717238 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.717170 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kpb4\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-kube-api-access-9kpb4\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.717460 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.717439 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-token\") pod \"istiod-openshift-gateway-7574459656-4fc9d\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.812672 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.812632 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:29:43.952527 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:43.952497 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d"] Jul 12 16:29:43.956063 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:43.956033 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff36d4f8_1e0a_43f0_ae3a_149e01522be6.slice/crio-f4c293328a76d404f061b6526bb9c8db4e19bd821d5abb432ab2ffc73a9eb7d0 WatchSource:0}: Error finding container f4c293328a76d404f061b6526bb9c8db4e19bd821d5abb432ab2ffc73a9eb7d0: Status 404 returned error can't find the container with id f4c293328a76d404f061b6526bb9c8db4e19bd821d5abb432ab2ffc73a9eb7d0 Jul 12 16:29:44.181136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:44.181097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" event={"ID":"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7","Type":"ContainerStarted","Data":"bccfc67a23fc6b6754a7fcfa4bab256f24825a052795f813e4566dbd4058df78"} Jul 12 16:29:44.182444 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:44.182409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" event={"ID":"ff36d4f8-1e0a-43f0-ae3a-149e01522be6","Type":"ContainerStarted","Data":"f4c293328a76d404f061b6526bb9c8db4e19bd821d5abb432ab2ffc73a9eb7d0"} Jul 12 16:29:44.940674 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:44.940638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:46.194548 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:46.194516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" event={"ID":"923f08c1-c7f8-4f05-b561-f38e4cfaa1e7","Type":"ContainerStarted","Data":"1e3066c1bb89137928fe04f6e5b468173b679bf580f0d9043d9dc0224e9d5823"} Jul 12 16:29:46.211423 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:46.211371 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-545795f48c-vwbng" podStartSLOduration=2.070939283 podStartE2EDuration="4.211353507s" totalImagesPullingTime="2.140414224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:29:46.210916657 +0000 UTC m=+284.980344567" watchObservedRunningTime="2026-07-12 16:29:46.211353507 +0000 UTC m=+284.980781402" Jul 12 16:29:47.348561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.348529 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:47.416895 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.416861 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl"] Jul 12 16:29:47.420118 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.420102 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:47.425073 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.425048 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2j5vq\"" Jul 12 16:29:47.425312 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.425294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 12 16:29:47.437231 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.437208 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl"] Jul 12 16:29:47.444724 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.444700 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd"] Jul 12 16:29:47.447935 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.447916 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:47.450960 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.450917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 12 16:29:47.451074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.451040 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-c8skt\"" Jul 12 16:29:47.451615 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.451573 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:47.461040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.461019 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd"] Jul 12 16:29:47.528630 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.528598 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:47.533759 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.533727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1c14d57-3002-459b-864c-c72ea58b2bc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-9s9vl\" (UID: \"e1c14d57-3002-459b-864c-c72ea58b2bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:47.533834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.533774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7pj\" (UniqueName: \"kubernetes.io/projected/5f3aeff8-d6d6-43a5-82bd-505568ed4002-kube-api-access-fv7pj\") pod \"llmisvc-controller-manager-777c6bd7d5-2ngrd\" (UID: \"5f3aeff8-d6d6-43a5-82bd-505568ed4002\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:47.533834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.533811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5lw\" (UniqueName: \"kubernetes.io/projected/e1c14d57-3002-459b-864c-c72ea58b2bc3-kube-api-access-5j5lw\") pod \"kserve-controller-manager-d6f4b65b8-9s9vl\" (UID: \"e1c14d57-3002-459b-864c-c72ea58b2bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:47.533912 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.533847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f3aeff8-d6d6-43a5-82bd-505568ed4002-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2ngrd\" (UID: \"5f3aeff8-d6d6-43a5-82bd-505568ed4002\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:47.634459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.634345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1c14d57-3002-459b-864c-c72ea58b2bc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-9s9vl\" (UID: \"e1c14d57-3002-459b-864c-c72ea58b2bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:47.634459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.634403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7pj\" (UniqueName: \"kubernetes.io/projected/5f3aeff8-d6d6-43a5-82bd-505568ed4002-kube-api-access-fv7pj\") pod \"llmisvc-controller-manager-777c6bd7d5-2ngrd\" (UID: \"5f3aeff8-d6d6-43a5-82bd-505568ed4002\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:47.634459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.634439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5lw\" (UniqueName: \"kubernetes.io/projected/e1c14d57-3002-459b-864c-c72ea58b2bc3-kube-api-access-5j5lw\") pod \"kserve-controller-manager-d6f4b65b8-9s9vl\" (UID: \"e1c14d57-3002-459b-864c-c72ea58b2bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:47.634459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.634463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f3aeff8-d6d6-43a5-82bd-505568ed4002-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2ngrd\" (UID: \"5f3aeff8-d6d6-43a5-82bd-505568ed4002\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:47.634787 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:29:47.634540 2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 12 16:29:47.634787 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:29:47.634625 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1c14d57-3002-459b-864c-c72ea58b2bc3-cert podName:e1c14d57-3002-459b-864c-c72ea58b2bc3 nodeName:}" failed. No retries permitted until 2026-07-12 16:29:48.134604862 +0000 UTC m=+286.904032738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e1c14d57-3002-459b-864c-c72ea58b2bc3-cert") pod "kserve-controller-manager-d6f4b65b8-9s9vl" (UID: "e1c14d57-3002-459b-864c-c72ea58b2bc3") : secret "kserve-webhook-server-cert" not found Jul 12 16:29:47.637071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.637040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f3aeff8-d6d6-43a5-82bd-505568ed4002-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2ngrd\" (UID: \"5f3aeff8-d6d6-43a5-82bd-505568ed4002\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:47.639825 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.639804 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:47.649647 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.649625 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5lw\" (UniqueName: \"kubernetes.io/projected/e1c14d57-3002-459b-864c-c72ea58b2bc3-kube-api-access-5j5lw\") pod \"kserve-controller-manager-d6f4b65b8-9s9vl\" (UID: \"e1c14d57-3002-459b-864c-c72ea58b2bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:47.649953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.649931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7pj\" (UniqueName: \"kubernetes.io/projected/5f3aeff8-d6d6-43a5-82bd-505568ed4002-kube-api-access-fv7pj\") pod \"llmisvc-controller-manager-777c6bd7d5-2ngrd\" (UID: \"5f3aeff8-d6d6-43a5-82bd-505568ed4002\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:47.760747 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.760711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:47.776939 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.776914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:47.897371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:47.897345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd"] Jul 12 16:29:47.899382 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:47.899358 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5f3aeff8_d6d6_43a5_82bd_505568ed4002.slice/crio-e80c835e767c0d9c236638ca3bf859d6085434897c15644c22b942898d342009 WatchSource:0}: Error finding container e80c835e767c0d9c236638ca3bf859d6085434897c15644c22b942898d342009: Status 404 returned error can't find the container with id e80c835e767c0d9c236638ca3bf859d6085434897c15644c22b942898d342009 Jul 12 16:29:48.139655 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:48.139611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1c14d57-3002-459b-864c-c72ea58b2bc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-9s9vl\" (UID: \"e1c14d57-3002-459b-864c-c72ea58b2bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:48.142063 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:48.142041 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1c14d57-3002-459b-864c-c72ea58b2bc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-9s9vl\" (UID: \"e1c14d57-3002-459b-864c-c72ea58b2bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:48.204239 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:48.204183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" event={"ID":"5f3aeff8-d6d6-43a5-82bd-505568ed4002","Type":"ContainerStarted","Data":"e80c835e767c0d9c236638ca3bf859d6085434897c15644c22b942898d342009"} Jul 12 16:29:48.330089 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:48.330059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:48.467386 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:48.467343 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl"] Jul 12 16:29:48.469547 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:48.469521 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c14d57_3002_459b_864c_c72ea58b2bc3.slice/crio-5d368cb856d135d3cdeb5ad07bd217c5f92e238e69a3f006cee0a1cb87ce0750 WatchSource:0}: Error finding container 5d368cb856d135d3cdeb5ad07bd217c5f92e238e69a3f006cee0a1cb87ce0750: Status 404 returned error can't find the container with id 5d368cb856d135d3cdeb5ad07bd217c5f92e238e69a3f006cee0a1cb87ce0750 Jul 12 16:29:48.636643 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:48.636609 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:48.748033 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:48.747935 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:48.861910 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:48.861874 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:49.212503 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:49.212457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" event={"ID":"e1c14d57-3002-459b-864c-c72ea58b2bc3","Type":"ContainerStarted","Data":"5d368cb856d135d3cdeb5ad07bd217c5f92e238e69a3f006cee0a1cb87ce0750"} Jul 12 16:29:50.203421 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:50.203389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:50.303836 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:50.303797 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:51.055495 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.055466 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8"] Jul 12 16:29:51.058797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.058775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.061644 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.061623 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7ds2p\"" Jul 12 16:29:51.072213 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.072168 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8"] Jul 12 16:29:51.096838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.096808 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d"] Jul 12 16:29:51.106386 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.106355 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:51.166684 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.166835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8057ab-8e02-4acf-8527-e8af17338ede-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.166835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.166835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.166835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.167067 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.167067 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.167067 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.167067 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.166953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpms\" (UniqueName: \"kubernetes.io/projected/cd8057ab-8e02-4acf-8527-e8af17338ede-kube-api-access-7tpms\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.217007 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.216976 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:51.226042 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.226010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" event={"ID":"5f3aeff8-d6d6-43a5-82bd-505568ed4002","Type":"ContainerStarted","Data":"7998170e0091b539f212528f802a5fd180f1d3e6d791f24a75d9f441c3200397"} Jul 12 16:29:51.226248 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.226075 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:29:51.227569 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.227544 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" event={"ID":"e1c14d57-3002-459b-864c-c72ea58b2bc3","Type":"ContainerStarted","Data":"b40492a9bda554532014a150fb20465f02c94b99235ca0fd56b168467cccb93b"} Jul 12 16:29:51.227685 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.227595 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:29:51.243156 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.243083 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" podStartSLOduration=1.47592856 podStartE2EDuration="4.243065983s" totalImagesPullingTime="2.767137423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:29:51.241700084 +0000 UTC m=+290.011127979" watchObservedRunningTime="2026-07-12 16:29:51.243065983 +0000 UTC m=+290.012493877" Jul 12 16:29:51.257653 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.257604 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" podStartSLOduration=2.058667406 podStartE2EDuration="4.257589544s" totalImagesPullingTime="2.198922138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:29:51.256663724 +0000 UTC m=+290.026091620" watchObservedRunningTime="2026-07-12 16:29:51.257589544 +0000 UTC m=+290.027017438" Jul 12 16:29:51.267726 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.267700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.267847 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.267734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.267847 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.267757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.267847 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.267786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.267985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268148 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268148 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpms\" (UniqueName: \"kubernetes.io/projected/cd8057ab-8e02-4acf-8527-e8af17338ede-kube-api-access-7tpms\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268338 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268144 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268338 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268338 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268216 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268338 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8057ab-8e02-4acf-8527-e8af17338ede-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268570 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268552 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268685 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268666 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.268907 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.268892 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8057ab-8e02-4acf-8527-e8af17338ede-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.270497 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.270451 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.270605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.270586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.276879 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.276856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8057ab-8e02-4acf-8527-e8af17338ede-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.276952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.276865 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpms\" (UniqueName: \"kubernetes.io/projected/cd8057ab-8e02-4acf-8527-e8af17338ede-kube-api-access-7tpms\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x7nw8\" (UID: \"cd8057ab-8e02-4acf-8527-e8af17338ede\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.369516 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.369428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:51.522221 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.522124 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8"] Jul 12 16:29:51.524339 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:29:51.524310 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8057ab_8e02_4acf_8527_e8af17338ede.slice/crio-3cdf064f998e655fe0270710c9ce2aa661b46cd6996c85fe4d9cc5025a964a79 WatchSource:0}: Error finding container 3cdf064f998e655fe0270710c9ce2aa661b46cd6996c85fe4d9cc5025a964a79: Status 404 returned error can't find the container with id 3cdf064f998e655fe0270710c9ce2aa661b46cd6996c85fe4d9cc5025a964a79 Jul 12 16:29:51.529176 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:51.529154 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:52.057510 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:52.057457 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:52.158671 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:52.158638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:29:52.172599 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:52.172573 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wccbh" Jul 12 16:29:52.174406 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:52.174385 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-x926v" Jul 12 16:29:52.236661 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:52.236624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" event={"ID":"cd8057ab-8e02-4acf-8527-e8af17338ede","Type":"ContainerStarted","Data":"3cdf064f998e655fe0270710c9ce2aa661b46cd6996c85fe4d9cc5025a964a79"} Jul 12 16:29:55.088474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:55.088440 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:29:55.088705 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:55.088510 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:29:55.088705 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:55.088541 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:29:55.252547 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:55.252508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" event={"ID":"cd8057ab-8e02-4acf-8527-e8af17338ede","Type":"ContainerStarted","Data":"5a9d5977d48b2ebd1230742bfd699d1d541f1e86c9187c55e0cf69131cf27adf"} Jul 12 16:29:55.273642 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:55.273589 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" podStartSLOduration=0.711904456 podStartE2EDuration="4.273573059s" totalImagesPullingTime="3.561668603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:29:55.272220109 +0000 UTC m=+294.041648026" watchObservedRunningTime="2026-07-12 16:29:55.273573059 +0000 UTC m=+294.043000953" Jul 12 16:29:55.369991 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:55.369911 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:56.375173 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:56.375144 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:57.263326 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:57.263298 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:29:57.264360 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:29:57.264339 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x7nw8" Jul 12 16:30:00.395223 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:00.395160 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:30:00.395492 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:00.395258 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:30:01.285693 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.285659 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" event={"ID":"ff36d4f8-1e0a-43f0-ae3a-149e01522be6","Type":"ContainerStarted","Data":"d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c"} Jul 12 16:30:01.285863 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.285726 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" podUID="ff36d4f8-1e0a-43f0-ae3a-149e01522be6" containerName="discovery" containerID="cri-o://d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c" gracePeriod=30 Jul 12 16:30:01.285863 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.285850 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:30:01.287769 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.287746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:30:01.310368 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.310319 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" podStartSLOduration=1.873448125 podStartE2EDuration="18.31030362s" totalImagesPullingTime="16.436855495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:30:01.308389668 +0000 UTC m=+300.077817592" watchObservedRunningTime="2026-07-12 16:30:01.31030362 +0000 UTC m=+300.079731515" Jul 12 16:30:01.534277 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.534251 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:30:01.663865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.662652 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-token\") pod \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " Jul 12 16:30:01.663865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.662714 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kpb4\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-kube-api-access-9kpb4\") pod \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " Jul 12 16:30:01.663865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.662749 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-kubeconfig\") pod \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " Jul 12 16:30:01.663865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.662775 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-cacerts\") pod \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " Jul 12 16:30:01.663865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.662822 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-dns-cert\") pod \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " Jul 12 16:30:01.663865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.662851 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-local-certs\") pod \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " Jul 12 16:30:01.663865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.662908 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-ca-configmap\") pod \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\" (UID: \"ff36d4f8-1e0a-43f0-ae3a-149e01522be6\") " Jul 12 16:30:01.663865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.663519 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ff36d4f8-1e0a-43f0-ae3a-149e01522be6" (UID: "ff36d4f8-1e0a-43f0-ae3a-149e01522be6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:30:01.666797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.666750 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-cacerts" (OuterVolumeSpecName: "cacerts") pod "ff36d4f8-1e0a-43f0-ae3a-149e01522be6" (UID: "ff36d4f8-1e0a-43f0-ae3a-149e01522be6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:30:01.666925 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.666855 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-kube-api-access-9kpb4" (OuterVolumeSpecName: "kube-api-access-9kpb4") pod "ff36d4f8-1e0a-43f0-ae3a-149e01522be6" (UID: "ff36d4f8-1e0a-43f0-ae3a-149e01522be6"). InnerVolumeSpecName "kube-api-access-9kpb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:30:01.667103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.667063 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ff36d4f8-1e0a-43f0-ae3a-149e01522be6" (UID: "ff36d4f8-1e0a-43f0-ae3a-149e01522be6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:30:01.667103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.667072 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-local-certs" (OuterVolumeSpecName: "local-certs") pod "ff36d4f8-1e0a-43f0-ae3a-149e01522be6" (UID: "ff36d4f8-1e0a-43f0-ae3a-149e01522be6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:01.667103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.667069 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff36d4f8-1e0a-43f0-ae3a-149e01522be6" (UID: "ff36d4f8-1e0a-43f0-ae3a-149e01522be6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:30:01.667315 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.667158 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ff36d4f8-1e0a-43f0-ae3a-149e01522be6" (UID: "ff36d4f8-1e0a-43f0-ae3a-149e01522be6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:30:01.715563 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.715522 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:30:01.716210 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.716173 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:30:01.722765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.722746 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 12 16:30:01.764479 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.764446 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-ca-configmap\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:01.764479 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.764474 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:01.764479 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.764484 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kpb4\" (UniqueName: \"kubernetes.io/projected/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-kube-api-access-9kpb4\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:01.764679 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.764494 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-kubeconfig\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:01.764679 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.764503 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-cacerts\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:01.764679 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.764512 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-istio-csr-dns-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:01.764679 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:01.764522 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff36d4f8-1e0a-43f0-ae3a-149e01522be6-local-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:02.291841 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.291803 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff36d4f8-1e0a-43f0-ae3a-149e01522be6" containerID="d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c" exitCode=0 Jul 12 16:30:02.292055 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.291848 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" Jul 12 16:30:02.292055 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.291870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" event={"ID":"ff36d4f8-1e0a-43f0-ae3a-149e01522be6","Type":"ContainerDied","Data":"d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c"} Jul 12 16:30:02.292055 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.291919 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d" event={"ID":"ff36d4f8-1e0a-43f0-ae3a-149e01522be6","Type":"ContainerDied","Data":"f4c293328a76d404f061b6526bb9c8db4e19bd821d5abb432ab2ffc73a9eb7d0"} Jul 12 16:30:02.292055 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.291945 2575 scope.go:117] "RemoveContainer" containerID="d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c" Jul 12 16:30:02.300537 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.300517 2575 scope.go:117] "RemoveContainer" containerID="d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c" Jul 12 16:30:02.300774 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:30:02.300756 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c\": container with ID starting with d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c not found: ID does not exist" containerID="d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c" Jul 12 16:30:02.300823 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.300785 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c"} err="failed to get container status \"d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c\": rpc error: code = NotFound desc = could not find container \"d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c\": container with ID starting with d2c060f95f3fc141496afd2ae4ff47ec42b0d06d8a5f8825e56be6a85588167c not found: ID does not exist" Jul 12 16:30:02.310498 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.310474 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d"] Jul 12 16:30:02.314071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:02.314050 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4fc9d"] Jul 12 16:30:03.874919 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:03.874884 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff36d4f8-1e0a-43f0-ae3a-149e01522be6" path="/var/lib/kubelet/pods/ff36d4f8-1e0a-43f0-ae3a-149e01522be6/volumes" Jul 12 16:30:22.238709 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:22.238677 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2ngrd" Jul 12 16:30:22.241675 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:22.241658 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9s9vl" Jul 12 16:30:38.171328 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.171248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:38.261692 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.261660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:38.297003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.296973 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6fc9"] Jul 12 16:30:38.297351 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.297338 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff36d4f8-1e0a-43f0-ae3a-149e01522be6" containerName="discovery" Jul 12 16:30:38.297351 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.297352 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff36d4f8-1e0a-43f0-ae3a-149e01522be6" containerName="discovery" Jul 12 16:30:38.297475 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.297427 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff36d4f8-1e0a-43f0-ae3a-149e01522be6" containerName="discovery" Jul 12 16:30:38.310772 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.310741 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" Jul 12 16:30:38.310922 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.310882 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6fc9"] Jul 12 16:30:38.313338 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.313317 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 12 16:30:38.313461 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.313407 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 12 16:30:38.314297 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.314280 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dtnc9\"" Jul 12 16:30:38.338478 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.338459 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:38.361458 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.361438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdwx\" (UniqueName: \"kubernetes.io/projected/5f6b53d1-5356-4b8b-ac32-970febbcc9ec-kube-api-access-pkdwx\") pod \"kuadrant-operator-catalog-d6fc9\" (UID: \"5f6b53d1-5356-4b8b-ac32-970febbcc9ec\") " pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" Jul 12 16:30:38.461990 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.461963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdwx\" (UniqueName: \"kubernetes.io/projected/5f6b53d1-5356-4b8b-ac32-970febbcc9ec-kube-api-access-pkdwx\") pod \"kuadrant-operator-catalog-d6fc9\" (UID: \"5f6b53d1-5356-4b8b-ac32-970febbcc9ec\") " pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" Jul 12 16:30:38.469650 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.469623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdwx\" (UniqueName: \"kubernetes.io/projected/5f6b53d1-5356-4b8b-ac32-970febbcc9ec-kube-api-access-pkdwx\") pod \"kuadrant-operator-catalog-d6fc9\" (UID: \"5f6b53d1-5356-4b8b-ac32-970febbcc9ec\") " pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" Jul 12 16:30:38.623464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.623429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" Jul 12 16:30:38.652222 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.652167 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6fc9"] Jul 12 16:30:38.755055 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.752799 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6fc9"] Jul 12 16:30:38.755884 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:30:38.755855 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6b53d1_5356_4b8b_ac32_970febbcc9ec.slice/crio-97e4b7900133f06339026b57cbfe571cb5f21cc48e69563aba96319b9bc6b534 WatchSource:0}: Error finding container 97e4b7900133f06339026b57cbfe571cb5f21cc48e69563aba96319b9bc6b534: Status 404 returned error can't find the container with id 97e4b7900133f06339026b57cbfe571cb5f21cc48e69563aba96319b9bc6b534 Jul 12 16:30:38.758051 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.758030 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 16:30:38.861888 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.861861 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dkpsc"] Jul 12 16:30:38.866452 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.866434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:38.873694 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.873671 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dkpsc"] Jul 12 16:30:38.966433 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:38.966394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgq9l\" (UniqueName: \"kubernetes.io/projected/49de7d40-8e73-4a34-b971-a3a95a42faab-kube-api-access-dgq9l\") pod \"kuadrant-operator-catalog-dkpsc\" (UID: \"49de7d40-8e73-4a34-b971-a3a95a42faab\") " pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:39.067037 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:39.066945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq9l\" (UniqueName: \"kubernetes.io/projected/49de7d40-8e73-4a34-b971-a3a95a42faab-kube-api-access-dgq9l\") pod \"kuadrant-operator-catalog-dkpsc\" (UID: \"49de7d40-8e73-4a34-b971-a3a95a42faab\") " pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:39.075223 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:39.075183 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq9l\" (UniqueName: \"kubernetes.io/projected/49de7d40-8e73-4a34-b971-a3a95a42faab-kube-api-access-dgq9l\") pod \"kuadrant-operator-catalog-dkpsc\" (UID: \"49de7d40-8e73-4a34-b971-a3a95a42faab\") " pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:39.084010 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:39.083990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:39.176763 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:39.176728 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:39.301152 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:39.301104 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:39.311143 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:39.311088 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dkpsc"] Jul 12 16:30:39.321338 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:30:39.321282 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49de7d40_8e73_4a34_b971_a3a95a42faab.slice/crio-97386b819b2815e8a8a8dfc1fec9f68f91ce35750f25d67e26bcfc2d9bfd7205 WatchSource:0}: Error finding container 97386b819b2815e8a8a8dfc1fec9f68f91ce35750f25d67e26bcfc2d9bfd7205: Status 404 returned error can't find the container with id 97386b819b2815e8a8a8dfc1fec9f68f91ce35750f25d67e26bcfc2d9bfd7205 Jul 12 16:30:39.463351 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:39.463310 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" event={"ID":"49de7d40-8e73-4a34-b971-a3a95a42faab","Type":"ContainerStarted","Data":"97386b819b2815e8a8a8dfc1fec9f68f91ce35750f25d67e26bcfc2d9bfd7205"} Jul 12 16:30:39.464727 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:39.464692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" event={"ID":"5f6b53d1-5356-4b8b-ac32-970febbcc9ec","Type":"ContainerStarted","Data":"97e4b7900133f06339026b57cbfe571cb5f21cc48e69563aba96319b9bc6b534"} Jul 12 16:30:41.476342 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.476295 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" event={"ID":"5f6b53d1-5356-4b8b-ac32-970febbcc9ec","Type":"ContainerStarted","Data":"97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964"} Jul 12 16:30:41.476787 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.476351 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" podUID="5f6b53d1-5356-4b8b-ac32-970febbcc9ec" containerName="registry-server" containerID="cri-o://97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964" gracePeriod=2 Jul 12 16:30:41.477848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.477821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" event={"ID":"49de7d40-8e73-4a34-b971-a3a95a42faab","Type":"ContainerStarted","Data":"b493ae35b2b92ba01f1a74e4f557e3b71d79e5c65e3f0cb7a6772c1686158ce2"} Jul 12 16:30:41.493243 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.493176 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" podStartSLOduration=1.531478098 podStartE2EDuration="3.493162425s" totalImagesPullingTime="1.961684327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:30:41.49201445 +0000 UTC m=+340.261442345" watchObservedRunningTime="2026-07-12 16:30:41.493162425 +0000 UTC m=+340.262590338" Jul 12 16:30:41.509406 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.509363 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" podStartSLOduration=2.111533917 podStartE2EDuration="3.509350706s" totalImagesPullingTime="1.397816789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:30:41.507286924 +0000 UTC m=+340.276714822" watchObservedRunningTime="2026-07-12 16:30:41.509350706 +0000 UTC m=+340.278778600" Jul 12 16:30:41.718146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.718122 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" Jul 12 16:30:41.791820 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.791746 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkdwx\" (UniqueName: \"kubernetes.io/projected/5f6b53d1-5356-4b8b-ac32-970febbcc9ec-kube-api-access-pkdwx\") pod \"5f6b53d1-5356-4b8b-ac32-970febbcc9ec\" (UID: \"5f6b53d1-5356-4b8b-ac32-970febbcc9ec\") " Jul 12 16:30:41.793780 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.793757 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6b53d1-5356-4b8b-ac32-970febbcc9ec-kube-api-access-pkdwx" (OuterVolumeSpecName: "kube-api-access-pkdwx") pod "5f6b53d1-5356-4b8b-ac32-970febbcc9ec" (UID: "5f6b53d1-5356-4b8b-ac32-970febbcc9ec"). InnerVolumeSpecName "kube-api-access-pkdwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:30:41.892941 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:41.892915 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkdwx\" (UniqueName: \"kubernetes.io/projected/5f6b53d1-5356-4b8b-ac32-970febbcc9ec-kube-api-access-pkdwx\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:42.483536 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.483450 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f6b53d1-5356-4b8b-ac32-970febbcc9ec" containerID="97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964" exitCode=0 Jul 12 16:30:42.483536 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.483494 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" Jul 12 16:30:42.483952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.483534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" event={"ID":"5f6b53d1-5356-4b8b-ac32-970febbcc9ec","Type":"ContainerDied","Data":"97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964"} Jul 12 16:30:42.483952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.483581 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d6fc9" event={"ID":"5f6b53d1-5356-4b8b-ac32-970febbcc9ec","Type":"ContainerDied","Data":"97e4b7900133f06339026b57cbfe571cb5f21cc48e69563aba96319b9bc6b534"} Jul 12 16:30:42.483952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.483608 2575 scope.go:117] "RemoveContainer" containerID="97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964" Jul 12 16:30:42.492804 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.492788 2575 scope.go:117] "RemoveContainer" containerID="97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964" Jul 12 16:30:42.493054 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:30:42.493035 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964\": container with ID starting with 97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964 not found: ID does not exist" containerID="97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964" Jul 12 16:30:42.493112 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.493061 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964"} err="failed to get container status \"97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964\": rpc error: code = NotFound desc = could not find container \"97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964\": container with ID starting with 97a7f45508a055492ecb9179e121a9de04fd20d9546bd507b2db80dd85b31964 not found: ID does not exist" Jul 12 16:30:42.501327 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.501304 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6fc9"] Jul 12 16:30:42.504632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.504612 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6fc9"] Jul 12 16:30:42.526151 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:42.526129 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:43.878949 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:43.878917 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f6b53d1-5356-4b8b-ac32-970febbcc9ec" path="/var/lib/kubelet/pods/5f6b53d1-5356-4b8b-ac32-970febbcc9ec/volumes" Jul 12 16:30:49.177026 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:49.176992 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:49.177498 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:49.177055 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:49.199308 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:49.199284 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:49.541375 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:49.541305 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-dkpsc" Jul 12 16:30:52.727511 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:52.727472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:53.720649 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.720570 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg"] Jul 12 16:30:53.720967 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.720954 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f6b53d1-5356-4b8b-ac32-970febbcc9ec" containerName="registry-server" Jul 12 16:30:53.720967 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.720969 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f6b53d1-5356-4b8b-ac32-970febbcc9ec" containerName="registry-server" Jul 12 16:30:53.721054 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.721035 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f6b53d1-5356-4b8b-ac32-970febbcc9ec" containerName="registry-server" Jul 12 16:30:53.724142 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.724117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.726865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.726847 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9n5ms\"" Jul 12 16:30:53.732766 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.732737 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg"] Jul 12 16:30:53.788139 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.788100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.788356 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.788159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7sq\" (UniqueName: \"kubernetes.io/projected/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-kube-api-access-5f7sq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.788356 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.788277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.889415 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.889376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7sq\" (UniqueName: \"kubernetes.io/projected/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-kube-api-access-5f7sq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.889580 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.889427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.889580 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.889485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.889853 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.889833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.889918 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.889868 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:53.897952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:53.897929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7sq\" (UniqueName: \"kubernetes.io/projected/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-kube-api-access-5f7sq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:54.033683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.033596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:54.163634 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.163606 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg"] Jul 12 16:30:54.165389 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:30:54.165364 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad922c3a_ed87_48e7_b3c7_1014cb8a1866.slice/crio-70ce9c69278093cc6833064353ccd7d38631e6289037438866de2806a5280b17 WatchSource:0}: Error finding container 70ce9c69278093cc6833064353ccd7d38631e6289037438866de2806a5280b17: Status 404 returned error can't find the container with id 70ce9c69278093cc6833064353ccd7d38631e6289037438866de2806a5280b17 Jul 12 16:30:54.305338 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.305263 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr"] Jul 12 16:30:54.308631 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.308609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.319515 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.319491 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr"] Jul 12 16:30:54.394131 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.394102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96pk\" (UniqueName: \"kubernetes.io/projected/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-kube-api-access-d96pk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.394297 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.394142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.394297 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.394255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.495651 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.495620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.495797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.495672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96pk\" (UniqueName: \"kubernetes.io/projected/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-kube-api-access-d96pk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.495797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.495706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.495985 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.495965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.496051 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.496032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.507720 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.507699 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96pk\" (UniqueName: \"kubernetes.io/projected/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-kube-api-access-d96pk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.543237 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.543177 2575 generic.go:356] "Generic (PLEG): container finished" podID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerID="0f92aa60e681369bd5cb78727ea7e2ac0c103fdacf402197b7a4accffbb439c4" exitCode=0 Jul 12 16:30:54.543363 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.543227 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" event={"ID":"ad922c3a-ed87-48e7-b3c7-1014cb8a1866","Type":"ContainerDied","Data":"0f92aa60e681369bd5cb78727ea7e2ac0c103fdacf402197b7a4accffbb439c4"} Jul 12 16:30:54.543363 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.543278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" event={"ID":"ad922c3a-ed87-48e7-b3c7-1014cb8a1866","Type":"ContainerStarted","Data":"70ce9c69278093cc6833064353ccd7d38631e6289037438866de2806a5280b17"} Jul 12 16:30:54.657685 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.657593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:54.699253 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.699221 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6"] Jul 12 16:30:54.703712 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.703687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.716267 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.715805 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6"] Jul 12 16:30:54.737884 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.737853 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:54.795457 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.795429 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr"] Jul 12 16:30:54.796948 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:30:54.796923 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b000cb_a0da_49ed_b7e5_2400ceeb46cb.slice/crio-30f0f57ca5bc5d2d853909a02a4dc020eab37f7b22d6b3d8c9e4112a1442bc88 WatchSource:0}: Error finding container 30f0f57ca5bc5d2d853909a02a4dc020eab37f7b22d6b3d8c9e4112a1442bc88: Status 404 returned error can't find the container with id 30f0f57ca5bc5d2d853909a02a4dc020eab37f7b22d6b3d8c9e4112a1442bc88 Jul 12 16:30:54.797217 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.797184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.797290 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.797237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.797387 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.797367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9cls\" (UniqueName: \"kubernetes.io/projected/072b95d8-8881-4843-9233-788aeea3645e-kube-api-access-g9cls\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.804829 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.804808 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:30:54.898548 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.898514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9cls\" (UniqueName: \"kubernetes.io/projected/072b95d8-8881-4843-9233-788aeea3645e-kube-api-access-g9cls\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.898710 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.898586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.898710 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.898628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.899019 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.899000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.899121 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.899098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:54.907646 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:54.907618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9cls\" (UniqueName: \"kubernetes.io/projected/072b95d8-8881-4843-9233-788aeea3645e-kube-api-access-g9cls\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:55.018248 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.018214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:55.102575 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.102534 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8"] Jul 12 16:30:55.106459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.106438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.117088 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.117006 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8"] Jul 12 16:30:55.201028 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.201001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.201125 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.201101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbw9t\" (UniqueName: \"kubernetes.io/projected/373f3c02-0b0a-44dc-910f-c05996b565ac-kube-api-access-tbw9t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.201221 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.201152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.269506 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.269482 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6"] Jul 12 16:30:55.270752 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:30:55.270731 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072b95d8_8881_4843_9233_788aeea3645e.slice/crio-6efec77d0ac6b4de821bd7e2c0f090edde4fad6e7ca8900353c896608c454456 WatchSource:0}: Error finding container 6efec77d0ac6b4de821bd7e2c0f090edde4fad6e7ca8900353c896608c454456: Status 404 returned error can't find the container with id 6efec77d0ac6b4de821bd7e2c0f090edde4fad6e7ca8900353c896608c454456 Jul 12 16:30:55.302114 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.302090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbw9t\" (UniqueName: \"kubernetes.io/projected/373f3c02-0b0a-44dc-910f-c05996b565ac-kube-api-access-tbw9t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.302221 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.302133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.302221 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.302165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.302560 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.302539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.302626 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.302559 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.309987 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.309965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbw9t\" (UniqueName: \"kubernetes.io/projected/373f3c02-0b0a-44dc-910f-c05996b565ac-kube-api-access-tbw9t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.418075 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.418047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:55.549496 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.549462 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8"] Jul 12 16:30:55.550756 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.550732 2575 generic.go:356] "Generic (PLEG): container finished" podID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerID="a3d43a995809b6e9feab27047cb31688064fca09761908fc172f7f314e2a720d" exitCode=0 Jul 12 16:30:55.550874 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.550817 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" event={"ID":"43b000cb-a0da-49ed-b7e5-2400ceeb46cb","Type":"ContainerDied","Data":"a3d43a995809b6e9feab27047cb31688064fca09761908fc172f7f314e2a720d"} Jul 12 16:30:55.550874 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.550865 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" event={"ID":"43b000cb-a0da-49ed-b7e5-2400ceeb46cb","Type":"ContainerStarted","Data":"30f0f57ca5bc5d2d853909a02a4dc020eab37f7b22d6b3d8c9e4112a1442bc88"} Jul 12 16:30:55.551355 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:30:55.551329 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373f3c02_0b0a_44dc_910f_c05996b565ac.slice/crio-d66b69340f2a90c6ae938cb86f251105d5e277442231cb98ea7165cd6c5c1e7d WatchSource:0}: Error finding container d66b69340f2a90c6ae938cb86f251105d5e277442231cb98ea7165cd6c5c1e7d: Status 404 returned error can't find the container with id d66b69340f2a90c6ae938cb86f251105d5e277442231cb98ea7165cd6c5c1e7d Jul 12 16:30:55.552596 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.552571 2575 generic.go:356] "Generic (PLEG): container finished" podID="072b95d8-8881-4843-9233-788aeea3645e" containerID="86f6e5f304a8fbf87803d5fb1ce39e6ebe44d057fa93594227e131b81f302e49" exitCode=0 Jul 12 16:30:55.552717 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.552661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" event={"ID":"072b95d8-8881-4843-9233-788aeea3645e","Type":"ContainerDied","Data":"86f6e5f304a8fbf87803d5fb1ce39e6ebe44d057fa93594227e131b81f302e49"} Jul 12 16:30:55.552784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.552713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" event={"ID":"072b95d8-8881-4843-9233-788aeea3645e","Type":"ContainerStarted","Data":"6efec77d0ac6b4de821bd7e2c0f090edde4fad6e7ca8900353c896608c454456"} Jul 12 16:30:55.554787 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.554765 2575 generic.go:356] "Generic (PLEG): container finished" podID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerID="584cd008cd57bedd6b46659e13a8a86b42ef99e862f9e2524afa463b9249dda4" exitCode=0 Jul 12 16:30:55.554857 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:55.554831 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" event={"ID":"ad922c3a-ed87-48e7-b3c7-1014cb8a1866","Type":"ContainerDied","Data":"584cd008cd57bedd6b46659e13a8a86b42ef99e862f9e2524afa463b9249dda4"} Jul 12 16:30:56.562139 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.562047 2575 generic.go:356] "Generic (PLEG): container finished" podID="072b95d8-8881-4843-9233-788aeea3645e" containerID="9911e5fbabd9cacea5e6766d29954e29d2e36700593c4cec338123addeb74a14" exitCode=0 Jul 12 16:30:56.562139 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.562128 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" event={"ID":"072b95d8-8881-4843-9233-788aeea3645e","Type":"ContainerDied","Data":"9911e5fbabd9cacea5e6766d29954e29d2e36700593c4cec338123addeb74a14"} Jul 12 16:30:56.564279 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.564252 2575 generic.go:356] "Generic (PLEG): container finished" podID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerID="90cf53d90ed3367000b95aa30fb18803cfc3e09737d35252fc625a5d564289b9" exitCode=0 Jul 12 16:30:56.564385 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.564329 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" event={"ID":"ad922c3a-ed87-48e7-b3c7-1014cb8a1866","Type":"ContainerDied","Data":"90cf53d90ed3367000b95aa30fb18803cfc3e09737d35252fc625a5d564289b9"} Jul 12 16:30:56.565793 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.565768 2575 generic.go:356] "Generic (PLEG): container finished" podID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerID="b863cf5d8f1c51d4db70a87888e6ca6a9df36693d93468390131dc1cc4ded0c5" exitCode=0 Jul 12 16:30:56.565904 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.565836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" event={"ID":"373f3c02-0b0a-44dc-910f-c05996b565ac","Type":"ContainerDied","Data":"b863cf5d8f1c51d4db70a87888e6ca6a9df36693d93468390131dc1cc4ded0c5"} Jul 12 16:30:56.565904 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.565875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" event={"ID":"373f3c02-0b0a-44dc-910f-c05996b565ac","Type":"ContainerStarted","Data":"d66b69340f2a90c6ae938cb86f251105d5e277442231cb98ea7165cd6c5c1e7d"} Jul 12 16:30:56.567740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.567719 2575 generic.go:356] "Generic (PLEG): container finished" podID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerID="3cd674c713b6c41aa13922950b53b0bf30918b34d15fd2fcfe33b54fb1a16aad" exitCode=0 Jul 12 16:30:56.567837 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:56.567802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" event={"ID":"43b000cb-a0da-49ed-b7e5-2400ceeb46cb","Type":"ContainerDied","Data":"3cd674c713b6c41aa13922950b53b0bf30918b34d15fd2fcfe33b54fb1a16aad"} Jul 12 16:30:57.574990 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.574955 2575 generic.go:356] "Generic (PLEG): container finished" podID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerID="626e221b8eb3f2b53d86b05e3809e021ad36213fdbc3104848e06087d4c600c9" exitCode=0 Jul 12 16:30:57.575404 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.575050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" event={"ID":"43b000cb-a0da-49ed-b7e5-2400ceeb46cb","Type":"ContainerDied","Data":"626e221b8eb3f2b53d86b05e3809e021ad36213fdbc3104848e06087d4c600c9"} Jul 12 16:30:57.577104 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.577084 2575 generic.go:356] "Generic (PLEG): container finished" podID="072b95d8-8881-4843-9233-788aeea3645e" containerID="69a69dacb6665c5a191a07fe4295da2babd7571f2a36348c9517315e08c0ce56" exitCode=0 Jul 12 16:30:57.577223 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.577114 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" event={"ID":"072b95d8-8881-4843-9233-788aeea3645e","Type":"ContainerDied","Data":"69a69dacb6665c5a191a07fe4295da2babd7571f2a36348c9517315e08c0ce56"} Jul 12 16:30:57.578953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.578929 2575 generic.go:356] "Generic (PLEG): container finished" podID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerID="a31b612e0c6ab9c87c1ccbf88e604ccc2ee173b67bfe3d485c5bce6de0a5ac97" exitCode=0 Jul 12 16:30:57.579061 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.578994 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" event={"ID":"373f3c02-0b0a-44dc-910f-c05996b565ac","Type":"ContainerDied","Data":"a31b612e0c6ab9c87c1ccbf88e604ccc2ee173b67bfe3d485c5bce6de0a5ac97"} Jul 12 16:30:57.712044 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.712021 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:57.823346 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.823313 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-bundle\") pod \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " Jul 12 16:30:57.823516 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.823411 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f7sq\" (UniqueName: \"kubernetes.io/projected/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-kube-api-access-5f7sq\") pod \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " Jul 12 16:30:57.823516 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.823456 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-util\") pod \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\" (UID: \"ad922c3a-ed87-48e7-b3c7-1014cb8a1866\") " Jul 12 16:30:57.823846 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.823816 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-bundle" (OuterVolumeSpecName: "bundle") pod "ad922c3a-ed87-48e7-b3c7-1014cb8a1866" (UID: "ad922c3a-ed87-48e7-b3c7-1014cb8a1866"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:57.825579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.825523 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-kube-api-access-5f7sq" (OuterVolumeSpecName: "kube-api-access-5f7sq") pod "ad922c3a-ed87-48e7-b3c7-1014cb8a1866" (UID: "ad922c3a-ed87-48e7-b3c7-1014cb8a1866"). InnerVolumeSpecName "kube-api-access-5f7sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:30:57.828962 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.828942 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-util" (OuterVolumeSpecName: "util") pod "ad922c3a-ed87-48e7-b3c7-1014cb8a1866" (UID: "ad922c3a-ed87-48e7-b3c7-1014cb8a1866"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:57.924687 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.924657 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7sq\" (UniqueName: \"kubernetes.io/projected/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-kube-api-access-5f7sq\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:57.924687 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.924682 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-util\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:57.924824 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:57.924693 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad922c3a-ed87-48e7-b3c7-1014cb8a1866-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:58.585954 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.585920 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" Jul 12 16:30:58.585954 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.585945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg" event={"ID":"ad922c3a-ed87-48e7-b3c7-1014cb8a1866","Type":"ContainerDied","Data":"70ce9c69278093cc6833064353ccd7d38631e6289037438866de2806a5280b17"} Jul 12 16:30:58.586402 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.585992 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70ce9c69278093cc6833064353ccd7d38631e6289037438866de2806a5280b17" Jul 12 16:30:58.588245 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.588219 2575 generic.go:356] "Generic (PLEG): container finished" podID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerID="35849b2b4e18efbe0c0cb19c8ec01a62536838795153fa43a5b7b30feb2a732b" exitCode=0 Jul 12 16:30:58.588363 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.588280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" event={"ID":"373f3c02-0b0a-44dc-910f-c05996b565ac","Type":"ContainerDied","Data":"35849b2b4e18efbe0c0cb19c8ec01a62536838795153fa43a5b7b30feb2a732b"} Jul 12 16:30:58.726134 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.726111 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:58.741120 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.741095 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:58.833103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.833063 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-bundle\") pod \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " Jul 12 16:30:58.833286 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.833159 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d96pk\" (UniqueName: \"kubernetes.io/projected/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-kube-api-access-d96pk\") pod \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " Jul 12 16:30:58.833286 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.833227 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-util\") pod \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\" (UID: \"43b000cb-a0da-49ed-b7e5-2400ceeb46cb\") " Jul 12 16:30:58.833286 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.833252 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-bundle\") pod \"072b95d8-8881-4843-9233-788aeea3645e\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " Jul 12 16:30:58.833417 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.833386 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-util\") pod \"072b95d8-8881-4843-9233-788aeea3645e\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " Jul 12 16:30:58.833462 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.833431 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9cls\" (UniqueName: \"kubernetes.io/projected/072b95d8-8881-4843-9233-788aeea3645e-kube-api-access-g9cls\") pod \"072b95d8-8881-4843-9233-788aeea3645e\" (UID: \"072b95d8-8881-4843-9233-788aeea3645e\") " Jul 12 16:30:58.833791 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.833762 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-bundle" (OuterVolumeSpecName: "bundle") pod "072b95d8-8881-4843-9233-788aeea3645e" (UID: "072b95d8-8881-4843-9233-788aeea3645e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:58.833949 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.833924 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-bundle" (OuterVolumeSpecName: "bundle") pod "43b000cb-a0da-49ed-b7e5-2400ceeb46cb" (UID: "43b000cb-a0da-49ed-b7e5-2400ceeb46cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:58.835392 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.835370 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-kube-api-access-d96pk" (OuterVolumeSpecName: "kube-api-access-d96pk") pod "43b000cb-a0da-49ed-b7e5-2400ceeb46cb" (UID: "43b000cb-a0da-49ed-b7e5-2400ceeb46cb"). InnerVolumeSpecName "kube-api-access-d96pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:30:58.835894 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.835872 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072b95d8-8881-4843-9233-788aeea3645e-kube-api-access-g9cls" (OuterVolumeSpecName: "kube-api-access-g9cls") pod "072b95d8-8881-4843-9233-788aeea3645e" (UID: "072b95d8-8881-4843-9233-788aeea3645e"). InnerVolumeSpecName "kube-api-access-g9cls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:30:58.839144 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.839075 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-util" (OuterVolumeSpecName: "util") pod "43b000cb-a0da-49ed-b7e5-2400ceeb46cb" (UID: "43b000cb-a0da-49ed-b7e5-2400ceeb46cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:58.839776 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.839755 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-util" (OuterVolumeSpecName: "util") pod "072b95d8-8881-4843-9233-788aeea3645e" (UID: "072b95d8-8881-4843-9233-788aeea3645e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:58.935155 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.935114 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96pk\" (UniqueName: \"kubernetes.io/projected/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-kube-api-access-d96pk\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:58.935155 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.935150 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-util\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:58.935155 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.935161 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:58.935458 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.935170 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/072b95d8-8881-4843-9233-788aeea3645e-util\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:58.935458 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.935178 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9cls\" (UniqueName: \"kubernetes.io/projected/072b95d8-8881-4843-9233-788aeea3645e-kube-api-access-g9cls\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:58.935458 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:58.935223 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43b000cb-a0da-49ed-b7e5-2400ceeb46cb-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:59.595300 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.595270 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" Jul 12 16:30:59.595300 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.595280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr" event={"ID":"43b000cb-a0da-49ed-b7e5-2400ceeb46cb","Type":"ContainerDied","Data":"30f0f57ca5bc5d2d853909a02a4dc020eab37f7b22d6b3d8c9e4112a1442bc88"} Jul 12 16:30:59.595778 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.595329 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f0f57ca5bc5d2d853909a02a4dc020eab37f7b22d6b3d8c9e4112a1442bc88" Jul 12 16:30:59.597133 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.597109 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" Jul 12 16:30:59.597133 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.597123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6" event={"ID":"072b95d8-8881-4843-9233-788aeea3645e","Type":"ContainerDied","Data":"6efec77d0ac6b4de821bd7e2c0f090edde4fad6e7ca8900353c896608c454456"} Jul 12 16:30:59.597299 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.597165 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6efec77d0ac6b4de821bd7e2c0f090edde4fad6e7ca8900353c896608c454456" Jul 12 16:30:59.729779 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.729753 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:30:59.843175 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.843137 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-util\") pod \"373f3c02-0b0a-44dc-910f-c05996b565ac\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " Jul 12 16:30:59.843175 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.843179 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-bundle\") pod \"373f3c02-0b0a-44dc-910f-c05996b565ac\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " Jul 12 16:30:59.843444 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.843252 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbw9t\" (UniqueName: \"kubernetes.io/projected/373f3c02-0b0a-44dc-910f-c05996b565ac-kube-api-access-tbw9t\") pod \"373f3c02-0b0a-44dc-910f-c05996b565ac\" (UID: \"373f3c02-0b0a-44dc-910f-c05996b565ac\") " Jul 12 16:30:59.843724 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.843695 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-bundle" (OuterVolumeSpecName: "bundle") pod "373f3c02-0b0a-44dc-910f-c05996b565ac" (UID: "373f3c02-0b0a-44dc-910f-c05996b565ac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:59.845435 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.845387 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373f3c02-0b0a-44dc-910f-c05996b565ac-kube-api-access-tbw9t" (OuterVolumeSpecName: "kube-api-access-tbw9t") pod "373f3c02-0b0a-44dc-910f-c05996b565ac" (UID: "373f3c02-0b0a-44dc-910f-c05996b565ac"). InnerVolumeSpecName "kube-api-access-tbw9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:30:59.849000 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.848967 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-util" (OuterVolumeSpecName: "util") pod "373f3c02-0b0a-44dc-910f-c05996b565ac" (UID: "373f3c02-0b0a-44dc-910f-c05996b565ac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:30:59.944901 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.944870 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:59.944901 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.944896 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbw9t\" (UniqueName: \"kubernetes.io/projected/373f3c02-0b0a-44dc-910f-c05996b565ac-kube-api-access-tbw9t\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:30:59.944901 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:30:59.944906 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373f3c02-0b0a-44dc-910f-c05996b565ac-util\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:00.604291 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:00.604255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" event={"ID":"373f3c02-0b0a-44dc-910f-c05996b565ac","Type":"ContainerDied","Data":"d66b69340f2a90c6ae938cb86f251105d5e277442231cb98ea7165cd6c5c1e7d"} Jul 12 16:31:00.604725 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:00.604308 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d66b69340f2a90c6ae938cb86f251105d5e277442231cb98ea7165cd6c5c1e7d" Jul 12 16:31:00.604725 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:00.604273 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8" Jul 12 16:31:07.247665 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:07.247622 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:09.951993 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.951959 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2"] Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952284 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerName="pull" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952295 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerName="pull" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952303 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072b95d8-8881-4843-9233-788aeea3645e" containerName="pull" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952308 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="072b95d8-8881-4843-9233-788aeea3645e" containerName="pull" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952317 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerName="extract" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952322 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerName="extract" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952331 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerName="extract" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952336 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerName="extract" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952344 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerName="util" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952348 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerName="util" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952354 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072b95d8-8881-4843-9233-788aeea3645e" containerName="util" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952358 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="072b95d8-8881-4843-9233-788aeea3645e" containerName="util" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952364 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerName="pull" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952369 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerName="pull" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952375 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerName="pull" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952380 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerName="pull" Jul 12 16:31:09.952378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952385 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerName="extract" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952390 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerName="extract" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952397 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerName="util" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952401 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerName="util" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952408 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerName="util" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952413 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerName="util" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952420 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072b95d8-8881-4843-9233-788aeea3645e" containerName="extract" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952425 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="072b95d8-8881-4843-9233-788aeea3645e" containerName="extract" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952475 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="072b95d8-8881-4843-9233-788aeea3645e" containerName="extract" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952483 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad922c3a-ed87-48e7-b3c7-1014cb8a1866" containerName="extract" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952489 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b000cb-a0da-49ed-b7e5-2400ceeb46cb" containerName="extract" Jul 12 16:31:09.952848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.952496 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="373f3c02-0b0a-44dc-910f-c05996b565ac" containerName="extract" Jul 12 16:31:09.958900 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.958882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:09.965323 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.965129 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vlhxh\"" Jul 12 16:31:09.972608 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:09.972590 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2"] Jul 12 16:31:10.136667 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:10.136631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2m7p\" (UniqueName: \"kubernetes.io/projected/92da4b99-1501-4784-9904-ac3dc444bf25-kube-api-access-z2m7p\") pod \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" (UID: \"92da4b99-1501-4784-9904-ac3dc444bf25\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:10.237819 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:10.237741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2m7p\" (UniqueName: \"kubernetes.io/projected/92da4b99-1501-4784-9904-ac3dc444bf25-kube-api-access-z2m7p\") pod \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" (UID: \"92da4b99-1501-4784-9904-ac3dc444bf25\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:10.254131 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:10.254106 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2m7p\" (UniqueName: \"kubernetes.io/projected/92da4b99-1501-4784-9904-ac3dc444bf25-kube-api-access-z2m7p\") pod \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" (UID: \"92da4b99-1501-4784-9904-ac3dc444bf25\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:10.268392 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:10.268370 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:10.411881 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:10.411856 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2"] Jul 12 16:31:10.413566 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:10.413538 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92da4b99_1501_4784_9904_ac3dc444bf25.slice/crio-b6b03a3f973c717a3d784f1ffdc22fc1776a19cd65b11cb81ef41901d5b91b88 WatchSource:0}: Error finding container b6b03a3f973c717a3d784f1ffdc22fc1776a19cd65b11cb81ef41901d5b91b88: Status 404 returned error can't find the container with id b6b03a3f973c717a3d784f1ffdc22fc1776a19cd65b11cb81ef41901d5b91b88 Jul 12 16:31:10.654820 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:10.654740 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" event={"ID":"92da4b99-1501-4784-9904-ac3dc444bf25","Type":"ContainerStarted","Data":"b6b03a3f973c717a3d784f1ffdc22fc1776a19cd65b11cb81ef41901d5b91b88"} Jul 12 16:31:10.898142 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:10.898075 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:12.665808 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:12.665773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" event={"ID":"92da4b99-1501-4784-9904-ac3dc444bf25","Type":"ContainerStarted","Data":"ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5"} Jul 12 16:31:12.666231 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:12.665905 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:12.685772 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:12.685720 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" podStartSLOduration=2.438675483 podStartE2EDuration="3.685705233s" totalImagesPullingTime="1.24702975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:12.683995722 +0000 UTC m=+371.453423617" watchObservedRunningTime="2026-07-12 16:31:12.685705233 +0000 UTC m=+371.455133127" Jul 12 16:31:15.004901 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.004861 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4"] Jul 12 16:31:15.010908 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.010885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:15.014119 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.014101 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rl62l\"" Jul 12 16:31:15.027354 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.027332 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4"] Jul 12 16:31:15.183068 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.183029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4btd\" (UniqueName: \"kubernetes.io/projected/9c898e98-3140-45fe-bc99-368e1b4885c9-kube-api-access-x4btd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" (UID: \"9c898e98-3140-45fe-bc99-368e1b4885c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:15.183237 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.183077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c898e98-3140-45fe-bc99-368e1b4885c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" (UID: \"9c898e98-3140-45fe-bc99-368e1b4885c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:15.283896 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.283821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4btd\" (UniqueName: \"kubernetes.io/projected/9c898e98-3140-45fe-bc99-368e1b4885c9-kube-api-access-x4btd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" (UID: \"9c898e98-3140-45fe-bc99-368e1b4885c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:15.283896 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.283861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c898e98-3140-45fe-bc99-368e1b4885c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" (UID: \"9c898e98-3140-45fe-bc99-368e1b4885c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:15.284273 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.284257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c898e98-3140-45fe-bc99-368e1b4885c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" (UID: \"9c898e98-3140-45fe-bc99-368e1b4885c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:15.300342 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.300319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4btd\" (UniqueName: \"kubernetes.io/projected/9c898e98-3140-45fe-bc99-368e1b4885c9-kube-api-access-x4btd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" (UID: \"9c898e98-3140-45fe-bc99-368e1b4885c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:15.321200 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.321160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:15.460798 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.460755 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4"] Jul 12 16:31:15.463937 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:15.463912 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c898e98_3140_45fe_bc99_368e1b4885c9.slice/crio-62230b47ffc622df94fb3524cede1a1ccde8579376711fb485177dbfd14193d4 WatchSource:0}: Error finding container 62230b47ffc622df94fb3524cede1a1ccde8579376711fb485177dbfd14193d4: Status 404 returned error can't find the container with id 62230b47ffc622df94fb3524cede1a1ccde8579376711fb485177dbfd14193d4 Jul 12 16:31:15.467057 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.466946 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:15.681676 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:15.681643 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" event={"ID":"9c898e98-3140-45fe-bc99-368e1b4885c9","Type":"ContainerStarted","Data":"62230b47ffc622df94fb3524cede1a1ccde8579376711fb485177dbfd14193d4"} Jul 12 16:31:16.066314 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:16.066238 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:16.433176 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:16.433144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:18.131730 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:18.131695 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:19.866388 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:19.866364 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:19.947371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:19.947339 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f6b6877c8-4jgqg"] Jul 12 16:31:19.950667 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:19.950652 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:19.971001 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:19.969942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f6b6877c8-4jgqg"] Jul 12 16:31:20.005011 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.004977 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:20.022226 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.022197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-service-ca\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.022382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.022234 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-oauth-config\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.022382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.022263 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-trusted-ca-bundle\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.022382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.022280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7gj\" (UniqueName: \"kubernetes.io/projected/d86f6fa7-2239-4cbc-93f6-db788a412edf-kube-api-access-ph7gj\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.022382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.022310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-config\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.022565 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.022398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-oauth-serving-cert\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.022565 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.022461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-serving-cert\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.105729 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.105610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:20.123431 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.123396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-oauth-serving-cert\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.123603 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.123456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-serving-cert\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.123603 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.123484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-service-ca\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.123603 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.123502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-oauth-config\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.123603 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.123531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-trusted-ca-bundle\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.123603 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.123556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7gj\" (UniqueName: \"kubernetes.io/projected/d86f6fa7-2239-4cbc-93f6-db788a412edf-kube-api-access-ph7gj\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.123603 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.123603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-config\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.124229 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.124171 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-oauth-serving-cert\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.124418 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.124234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-service-ca\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.124549 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.124308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-config\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.124948 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.124926 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86f6fa7-2239-4cbc-93f6-db788a412edf-trusted-ca-bundle\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.125961 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.125937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-oauth-config\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.126394 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.126374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86f6fa7-2239-4cbc-93f6-db788a412edf-console-serving-cert\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.132880 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.132859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7gj\" (UniqueName: \"kubernetes.io/projected/d86f6fa7-2239-4cbc-93f6-db788a412edf-kube-api-access-ph7gj\") pod \"console-f6b6877c8-4jgqg\" (UID: \"d86f6fa7-2239-4cbc-93f6-db788a412edf\") " pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.259626 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.259588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:20.397465 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.397441 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f6b6877c8-4jgqg"] Jul 12 16:31:20.399087 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:20.399055 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86f6fa7_2239_4cbc_93f6_db788a412edf.slice/crio-89c53daf96126c23e4c29d2d2425dffb7cb34ff8906303578d2dd9e6c271bc9c WatchSource:0}: Error finding container 89c53daf96126c23e4c29d2d2425dffb7cb34ff8906303578d2dd9e6c271bc9c: Status 404 returned error can't find the container with id 89c53daf96126c23e4c29d2d2425dffb7cb34ff8906303578d2dd9e6c271bc9c Jul 12 16:31:20.718761 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.718721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" event={"ID":"9c898e98-3140-45fe-bc99-368e1b4885c9","Type":"ContainerStarted","Data":"4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af"} Jul 12 16:31:20.718928 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.718890 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:20.720445 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.720421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6b6877c8-4jgqg" event={"ID":"d86f6fa7-2239-4cbc-93f6-db788a412edf","Type":"ContainerStarted","Data":"1855fbe3683c7c8432d62b19199682b46c8e7d31f1e2bc42683c3b541e00c729"} Jul 12 16:31:20.720587 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.720453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6b6877c8-4jgqg" event={"ID":"d86f6fa7-2239-4cbc-93f6-db788a412edf","Type":"ContainerStarted","Data":"89c53daf96126c23e4c29d2d2425dffb7cb34ff8906303578d2dd9e6c271bc9c"} Jul 12 16:31:20.762745 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.762712 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:20.792029 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.791975 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" podStartSLOduration=2.419997921 podStartE2EDuration="6.791959895s" totalImagesPullingTime="4.371961974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:20.750679451 +0000 UTC m=+379.520107346" watchObservedRunningTime="2026-07-12 16:31:20.791959895 +0000 UTC m=+379.561387790" Jul 12 16:31:20.793233 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.793201 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f6b6877c8-4jgqg" podStartSLOduration=1.793179254 podStartE2EDuration="1.793179254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:20.791649746 +0000 UTC m=+379.561077659" watchObservedRunningTime="2026-07-12 16:31:20.793179254 +0000 UTC m=+379.562607148" Jul 12 16:31:20.843988 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:20.843954 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:22.106505 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.106475 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx"] Jul 12 16:31:22.110058 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.110037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" Jul 12 16:31:22.113924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.113896 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 12 16:31:22.114028 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.113953 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-p7x6p\"" Jul 12 16:31:22.132023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.132002 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx"] Jul 12 16:31:22.139378 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.139352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktkj\" (UniqueName: \"kubernetes.io/projected/b4ab23b1-52f4-46d2-bbdf-256e4929b8a0-kube-api-access-4ktkj\") pod \"dns-operator-controller-manager-864dd78c4c-pmbpx\" (UID: \"b4ab23b1-52f4-46d2-bbdf-256e4929b8a0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" Jul 12 16:31:22.240550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.240515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktkj\" (UniqueName: \"kubernetes.io/projected/b4ab23b1-52f4-46d2-bbdf-256e4929b8a0-kube-api-access-4ktkj\") pod \"dns-operator-controller-manager-864dd78c4c-pmbpx\" (UID: \"b4ab23b1-52f4-46d2-bbdf-256e4929b8a0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" Jul 12 16:31:22.248500 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.248474 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktkj\" (UniqueName: \"kubernetes.io/projected/b4ab23b1-52f4-46d2-bbdf-256e4929b8a0-kube-api-access-4ktkj\") pod \"dns-operator-controller-manager-864dd78c4c-pmbpx\" (UID: \"b4ab23b1-52f4-46d2-bbdf-256e4929b8a0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" Jul 12 16:31:22.420800 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.420712 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" Jul 12 16:31:22.549614 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.549591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx"] Jul 12 16:31:22.551818 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:22.551792 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ab23b1_52f4_46d2_bbdf_256e4929b8a0.slice/crio-51a182f38b5b3d97a4a5a91b9e071769d710b52aa7c5de2b5e0c33f4606d8963 WatchSource:0}: Error finding container 51a182f38b5b3d97a4a5a91b9e071769d710b52aa7c5de2b5e0c33f4606d8963: Status 404 returned error can't find the container with id 51a182f38b5b3d97a4a5a91b9e071769d710b52aa7c5de2b5e0c33f4606d8963 Jul 12 16:31:22.623109 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.623076 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs"] Jul 12 16:31:22.627779 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.627760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.630945 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.630925 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 12 16:31:22.631071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.630965 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 12 16:31:22.631071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.630986 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9n5ms\"" Jul 12 16:31:22.637578 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.637555 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs"] Jul 12 16:31:22.644457 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.644435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a386d73-e480-4fee-a740-19265d43ccb5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.644553 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.644483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a386d73-e480-4fee-a740-19265d43ccb5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.644553 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.644501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xkc\" (UniqueName: \"kubernetes.io/projected/3a386d73-e480-4fee-a740-19265d43ccb5-kube-api-access-g7xkc\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.657344 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.657318 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:22.739100 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.739060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" event={"ID":"b4ab23b1-52f4-46d2-bbdf-256e4929b8a0","Type":"ContainerStarted","Data":"51a182f38b5b3d97a4a5a91b9e071769d710b52aa7c5de2b5e0c33f4606d8963"} Jul 12 16:31:22.745564 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.745544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a386d73-e480-4fee-a740-19265d43ccb5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.745640 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.745572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xkc\" (UniqueName: \"kubernetes.io/projected/3a386d73-e480-4fee-a740-19265d43ccb5-kube-api-access-g7xkc\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.745677 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.745645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a386d73-e480-4fee-a740-19265d43ccb5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.745755 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:22.745741 2575 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 12 16:31:22.745815 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:22.745806 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a386d73-e480-4fee-a740-19265d43ccb5-plugin-serving-cert podName:3a386d73-e480-4fee-a740-19265d43ccb5 nodeName:}" failed. No retries permitted until 2026-07-12 16:31:23.245786039 +0000 UTC m=+382.015213914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3a386d73-e480-4fee-a740-19265d43ccb5-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-k44fs" (UID: "3a386d73-e480-4fee-a740-19265d43ccb5") : secret "plugin-serving-cert" not found Jul 12 16:31:22.746244 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.746225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a386d73-e480-4fee-a740-19265d43ccb5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.756830 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.756798 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:22.757823 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.757797 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xkc\" (UniqueName: \"kubernetes.io/projected/3a386d73-e480-4fee-a740-19265d43ccb5-kube-api-access-g7xkc\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:22.863540 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:22.863507 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:23.250551 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:23.250518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a386d73-e480-4fee-a740-19265d43ccb5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:23.252994 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:23.252967 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a386d73-e480-4fee-a740-19265d43ccb5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k44fs\" (UID: \"3a386d73-e480-4fee-a740-19265d43ccb5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:23.537935 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:23.537855 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" Jul 12 16:31:23.672717 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:23.672688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:23.689798 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:23.689773 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs"] Jul 12 16:31:23.979635 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:23.979605 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a386d73_e480_4fee_a740_19265d43ccb5.slice/crio-c9f2a22c83bb9942a1462fd4343c3be800bfeab7b20b2f29bee05a28c4a8bf79 WatchSource:0}: Error finding container c9f2a22c83bb9942a1462fd4343c3be800bfeab7b20b2f29bee05a28c4a8bf79: Status 404 returned error can't find the container with id c9f2a22c83bb9942a1462fd4343c3be800bfeab7b20b2f29bee05a28c4a8bf79 Jul 12 16:31:24.054734 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:24.054707 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:24.161700 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:24.161666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:24.756408 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:24.756369 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" event={"ID":"b4ab23b1-52f4-46d2-bbdf-256e4929b8a0","Type":"ContainerStarted","Data":"a3c693ab5f3c4561a143145abee4c9610e0a0c30d193432a42580e87e4a7a450"} Jul 12 16:31:24.756988 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:24.756570 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" Jul 12 16:31:24.758702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:24.758673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" event={"ID":"3a386d73-e480-4fee-a740-19265d43ccb5","Type":"ContainerStarted","Data":"c9f2a22c83bb9942a1462fd4343c3be800bfeab7b20b2f29bee05a28c4a8bf79"} Jul 12 16:31:24.775158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:24.775110 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" podStartSLOduration=1.3001759480000001 podStartE2EDuration="2.775096209s" totalImagesPullingTime="1.474920261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:24.773319312 +0000 UTC m=+383.542747228" watchObservedRunningTime="2026-07-12 16:31:24.775096209 +0000 UTC m=+383.544524166" Jul 12 16:31:27.447654 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:27.447604 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:27.506404 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:27.506372 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:30.260047 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:30.260014 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:30.260557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:30.260089 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:30.265155 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:30.265132 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:30.811272 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:30.811234 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f6b6877c8-4jgqg" Jul 12 16:31:30.868851 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:30.868808 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55dc84b75c-2tbh6"] Jul 12 16:31:30.988558 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:30.988521 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:31.089738 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:31.089658 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:31.191307 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:31.191272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:31.729113 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:31.729077 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:33.111705 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.111658 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn"] Jul 12 16:31:33.132241 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.132175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.137894 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.137866 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn"] Jul 12 16:31:33.245110 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.245070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08934781-961f-4082-a3c7-3a64358862f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" (UID: \"08934781-961f-4082-a3c7-3a64358862f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.245312 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.245117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbw9h\" (UniqueName: \"kubernetes.io/projected/08934781-961f-4082-a3c7-3a64358862f8-kube-api-access-fbw9h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" (UID: \"08934781-961f-4082-a3c7-3a64358862f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.307055 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.306960 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn"] Jul 12 16:31:33.307351 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.307324 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-fbw9h], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" podUID="08934781-961f-4082-a3c7-3a64358862f8" Jul 12 16:31:33.345864 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.345810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08934781-961f-4082-a3c7-3a64358862f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" (UID: \"08934781-961f-4082-a3c7-3a64358862f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.345864 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.345861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbw9h\" (UniqueName: \"kubernetes.io/projected/08934781-961f-4082-a3c7-3a64358862f8-kube-api-access-fbw9h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" (UID: \"08934781-961f-4082-a3c7-3a64358862f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.346284 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.346252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08934781-961f-4082-a3c7-3a64358862f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" (UID: \"08934781-961f-4082-a3c7-3a64358862f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.363010 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.362943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbw9h\" (UniqueName: \"kubernetes.io/projected/08934781-961f-4082-a3c7-3a64358862f8-kube-api-access-fbw9h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" (UID: \"08934781-961f-4082-a3c7-3a64358862f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.620722 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.620643 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn"] Jul 12 16:31:33.637542 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.637506 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn"] Jul 12 16:31:33.658379 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.658345 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4"] Jul 12 16:31:33.658655 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.658625 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" containerName="manager" containerID="cri-o://4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af" gracePeriod=2 Jul 12 16:31:33.661453 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.661426 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt"] Jul 12 16:31:33.673845 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.673292 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4"] Jul 12 16:31:33.673845 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.673419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:33.676522 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.676490 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:33.689783 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.689755 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2"] Jul 12 16:31:33.690011 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.689978 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" containerName="manager" containerID="cri-o://ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5" gracePeriod=2 Jul 12 16:31:33.692906 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.692879 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt"] Jul 12 16:31:33.705386 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.705117 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l"] Jul 12 16:31:33.705999 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.705711 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" containerName="manager" Jul 12 16:31:33.705999 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.705735 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" containerName="manager" Jul 12 16:31:33.705999 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.705851 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" containerName="manager" Jul 12 16:31:33.711485 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.711461 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2"] Jul 12 16:31:33.711611 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.711596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:33.727787 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.727755 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:33.734050 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.734017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l"] Jul 12 16:31:33.748928 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.748897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3094a87b-2bcb-4719-9ecc-d6118a842f10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7dnjt\" (UID: \"3094a87b-2bcb-4719-9ecc-d6118a842f10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:33.749068 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.748996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4rv\" (UniqueName: \"kubernetes.io/projected/3094a87b-2bcb-4719-9ecc-d6118a842f10-kube-api-access-gb4rv\") pod \"kuadrant-operator-controller-manager-55467d9f7-7dnjt\" (UID: \"3094a87b-2bcb-4719-9ecc-d6118a842f10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:33.749068 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.749028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5f861b5-1052-46fd-affe-31395aed358b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcv8l\" (UID: \"e5f861b5-1052-46fd-affe-31395aed358b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:33.749218 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.749094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/e5f861b5-1052-46fd-affe-31395aed358b-kube-api-access-rdn65\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcv8l\" (UID: \"e5f861b5-1052-46fd-affe-31395aed358b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:33.767069 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.767041 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6"] Jul 12 16:31:33.767676 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.767656 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" containerName="manager" Jul 12 16:31:33.767769 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.767680 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" containerName="manager" Jul 12 16:31:33.767821 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.767795 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" containerName="manager" Jul 12 16:31:33.772759 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.772739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" Jul 12 16:31:33.779940 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.779908 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:33.784261 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.784228 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:33.788053 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.788022 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:33.814203 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.814035 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6"] Jul 12 16:31:33.826147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.826120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.835616 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.835592 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.836908 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.836873 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:33.839480 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.839449 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:33.841923 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.841887 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="08934781-961f-4082-a3c7-3a64358862f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.843894 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.843862 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:33.846322 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.846296 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:33.848455 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:33.848434 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="08934781-961f-4082-a3c7-3a64358862f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:33.849835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.849809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/e5f861b5-1052-46fd-affe-31395aed358b-kube-api-access-rdn65\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcv8l\" (UID: \"e5f861b5-1052-46fd-affe-31395aed358b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:33.849947 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.849901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7nw\" (UniqueName: \"kubernetes.io/projected/afe1243b-90d8-494c-a826-8f974945c7c2-kube-api-access-xq7nw\") pod \"limitador-operator-controller-manager-859d7b78bd-7wth6\" (UID: \"afe1243b-90d8-494c-a826-8f974945c7c2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" Jul 12 16:31:33.850049 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.850008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3094a87b-2bcb-4719-9ecc-d6118a842f10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7dnjt\" (UID: \"3094a87b-2bcb-4719-9ecc-d6118a842f10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:33.850134 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.850086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4rv\" (UniqueName: \"kubernetes.io/projected/3094a87b-2bcb-4719-9ecc-d6118a842f10-kube-api-access-gb4rv\") pod \"kuadrant-operator-controller-manager-55467d9f7-7dnjt\" (UID: \"3094a87b-2bcb-4719-9ecc-d6118a842f10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:33.850134 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.850105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5f861b5-1052-46fd-affe-31395aed358b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcv8l\" (UID: \"e5f861b5-1052-46fd-affe-31395aed358b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:33.850414 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.850390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3094a87b-2bcb-4719-9ecc-d6118a842f10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7dnjt\" (UID: \"3094a87b-2bcb-4719-9ecc-d6118a842f10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:33.850549 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.850529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5f861b5-1052-46fd-affe-31395aed358b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcv8l\" (UID: \"e5f861b5-1052-46fd-affe-31395aed358b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:33.863474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.863426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/e5f861b5-1052-46fd-affe-31395aed358b-kube-api-access-rdn65\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcv8l\" (UID: \"e5f861b5-1052-46fd-affe-31395aed358b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:33.863474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.863446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4rv\" (UniqueName: \"kubernetes.io/projected/3094a87b-2bcb-4719-9ecc-d6118a842f10-kube-api-access-gb4rv\") pod \"kuadrant-operator-controller-manager-55467d9f7-7dnjt\" (UID: \"3094a87b-2bcb-4719-9ecc-d6118a842f10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:33.950862 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.950829 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbw9h\" (UniqueName: \"kubernetes.io/projected/08934781-961f-4082-a3c7-3a64358862f8-kube-api-access-fbw9h\") pod \"08934781-961f-4082-a3c7-3a64358862f8\" (UID: \"08934781-961f-4082-a3c7-3a64358862f8\") " Jul 12 16:31:33.951053 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.950939 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08934781-961f-4082-a3c7-3a64358862f8-extensions-socket-volume\") pod \"08934781-961f-4082-a3c7-3a64358862f8\" (UID: \"08934781-961f-4082-a3c7-3a64358862f8\") " Jul 12 16:31:33.951142 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.951119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7nw\" (UniqueName: \"kubernetes.io/projected/afe1243b-90d8-494c-a826-8f974945c7c2-kube-api-access-xq7nw\") pod \"limitador-operator-controller-manager-859d7b78bd-7wth6\" (UID: \"afe1243b-90d8-494c-a826-8f974945c7c2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" Jul 12 16:31:33.951269 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.951240 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08934781-961f-4082-a3c7-3a64358862f8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "08934781-961f-4082-a3c7-3a64358862f8" (UID: "08934781-961f-4082-a3c7-3a64358862f8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:31:33.953624 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.953590 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08934781-961f-4082-a3c7-3a64358862f8-kube-api-access-fbw9h" (OuterVolumeSpecName: "kube-api-access-fbw9h") pod "08934781-961f-4082-a3c7-3a64358862f8" (UID: "08934781-961f-4082-a3c7-3a64358862f8"). InnerVolumeSpecName "kube-api-access-fbw9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:31:33.964327 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:33.964303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7nw\" (UniqueName: \"kubernetes.io/projected/afe1243b-90d8-494c-a826-8f974945c7c2-kube-api-access-xq7nw\") pod \"limitador-operator-controller-manager-859d7b78bd-7wth6\" (UID: \"afe1243b-90d8-494c-a826-8f974945c7c2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" Jul 12 16:31:34.052582 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:34.052546 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08934781-961f-4082-a3c7-3a64358862f8-extensions-socket-volume\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:34.052582 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:34.052582 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbw9h\" (UniqueName: \"kubernetes.io/projected/08934781-961f-4082-a3c7-3a64358862f8-kube-api-access-fbw9h\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:34.092417 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:34.092382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:34.101172 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:34.101144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:34.108446 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:34.108421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" Jul 12 16:31:34.832058 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:34.832025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:34.842029 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:34.841991 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="08934781-961f-4082-a3c7-3a64358862f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:35.769669 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:35.769636 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pmbpx" Jul 12 16:31:35.772140 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:35.772098 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kmhnn\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="08934781-961f-4082-a3c7-3a64358862f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kmhnn" Jul 12 16:31:35.886364 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:35.886329 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08934781-961f-4082-a3c7-3a64358862f8" path="/var/lib/kubelet/pods/08934781-961f-4082-a3c7-3a64358862f8/volumes" Jul 12 16:31:36.134498 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:36.133944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:36.228791 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:36.228755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:38.505927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:38.505895 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:38.569481 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:38.569452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:42.769467 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:42.769423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:42.869511 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:42.869474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:45.268436 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.268400 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:45.374206 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.372280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:45.501716 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.501551 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:45.504571 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:45.503876 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:45.516627 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.514460 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:45.516627 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:45.516268 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3094a87b_2bcb_4719_9ecc_d6118a842f10.slice/crio-0dd1a35eb4221e0894453b214f028577573cf637c2c16f1e5e5ba9ac4e23cd2e WatchSource:0}: Error finding container 0dd1a35eb4221e0894453b214f028577573cf637c2c16f1e5e5ba9ac4e23cd2e: Status 404 returned error can't find the container with id 0dd1a35eb4221e0894453b214f028577573cf637c2c16f1e5e5ba9ac4e23cd2e Jul 12 16:31:45.516627 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.516474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt"] Jul 12 16:31:45.517067 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:45.516845 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:45.519339 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:45.519129 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:45.539795 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.539764 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6"] Jul 12 16:31:45.541624 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:45.541555 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe1243b_90d8_494c_a826_8f974945c7c2.slice/crio-20fcc2ad2bd20cbbe96202be56d87576c5738c75583d197404f875db260fd677 WatchSource:0}: Error finding container 20fcc2ad2bd20cbbe96202be56d87576c5738c75583d197404f875db260fd677: Status 404 returned error can't find the container with id 20fcc2ad2bd20cbbe96202be56d87576c5738c75583d197404f875db260fd677 Jul 12 16:31:45.558895 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.558873 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2m7p\" (UniqueName: \"kubernetes.io/projected/92da4b99-1501-4784-9904-ac3dc444bf25-kube-api-access-z2m7p\") pod \"92da4b99-1501-4784-9904-ac3dc444bf25\" (UID: \"92da4b99-1501-4784-9904-ac3dc444bf25\") " Jul 12 16:31:45.560844 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.560815 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92da4b99-1501-4784-9904-ac3dc444bf25-kube-api-access-z2m7p" (OuterVolumeSpecName: "kube-api-access-z2m7p") pod "92da4b99-1501-4784-9904-ac3dc444bf25" (UID: "92da4b99-1501-4784-9904-ac3dc444bf25"). InnerVolumeSpecName "kube-api-access-z2m7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:31:45.565983 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.565962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l"] Jul 12 16:31:45.568230 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:45.568202 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f861b5_1052_46fd_affe_31395aed358b.slice/crio-95379a3acd84aafc1c38d41fb0f4b4d78fd0953989cc73fafc5aabc5cbbecf07 WatchSource:0}: Error finding container 95379a3acd84aafc1c38d41fb0f4b4d78fd0953989cc73fafc5aabc5cbbecf07: Status 404 returned error can't find the container with id 95379a3acd84aafc1c38d41fb0f4b4d78fd0953989cc73fafc5aabc5cbbecf07 Jul 12 16:31:45.660485 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.660455 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c898e98-3140-45fe-bc99-368e1b4885c9-extensions-socket-volume\") pod \"9c898e98-3140-45fe-bc99-368e1b4885c9\" (UID: \"9c898e98-3140-45fe-bc99-368e1b4885c9\") " Jul 12 16:31:45.660674 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.660545 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4btd\" (UniqueName: \"kubernetes.io/projected/9c898e98-3140-45fe-bc99-368e1b4885c9-kube-api-access-x4btd\") pod \"9c898e98-3140-45fe-bc99-368e1b4885c9\" (UID: \"9c898e98-3140-45fe-bc99-368e1b4885c9\") " Jul 12 16:31:45.660797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.660780 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2m7p\" (UniqueName: \"kubernetes.io/projected/92da4b99-1501-4784-9904-ac3dc444bf25-kube-api-access-z2m7p\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:45.660970 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.660952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c898e98-3140-45fe-bc99-368e1b4885c9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9c898e98-3140-45fe-bc99-368e1b4885c9" (UID: "9c898e98-3140-45fe-bc99-368e1b4885c9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:31:45.662548 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.662524 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c898e98-3140-45fe-bc99-368e1b4885c9-kube-api-access-x4btd" (OuterVolumeSpecName: "kube-api-access-x4btd") pod "9c898e98-3140-45fe-bc99-368e1b4885c9" (UID: "9c898e98-3140-45fe-bc99-368e1b4885c9"). InnerVolumeSpecName "kube-api-access-x4btd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:31:45.761468 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.761426 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4btd\" (UniqueName: \"kubernetes.io/projected/9c898e98-3140-45fe-bc99-368e1b4885c9-kube-api-access-x4btd\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:45.761468 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.761466 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c898e98-3140-45fe-bc99-368e1b4885c9-extensions-socket-volume\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:45.875913 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.875828 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" path="/var/lib/kubelet/pods/92da4b99-1501-4784-9904-ac3dc444bf25/volumes" Jul 12 16:31:45.876183 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.876171 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" path="/var/lib/kubelet/pods/9c898e98-3140-45fe-bc99-368e1b4885c9/volumes" Jul 12 16:31:45.911847 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.911807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" event={"ID":"e5f861b5-1052-46fd-affe-31395aed358b","Type":"ContainerStarted","Data":"20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618"} Jul 12 16:31:45.912174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.911976 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" event={"ID":"e5f861b5-1052-46fd-affe-31395aed358b","Type":"ContainerStarted","Data":"95379a3acd84aafc1c38d41fb0f4b4d78fd0953989cc73fafc5aabc5cbbecf07"} Jul 12 16:31:45.912174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.912048 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:45.914370 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.914337 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" event={"ID":"afe1243b-90d8-494c-a826-8f974945c7c2","Type":"ContainerStarted","Data":"438a7fcd87cf0016bb8b7c5e6fa3d886e36ba8a981a2ef331ade12b0b869baea"} Jul 12 16:31:45.914505 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.914394 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" event={"ID":"afe1243b-90d8-494c-a826-8f974945c7c2","Type":"ContainerStarted","Data":"20fcc2ad2bd20cbbe96202be56d87576c5738c75583d197404f875db260fd677"} Jul 12 16:31:45.914571 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.914509 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" Jul 12 16:31:45.915912 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.915887 2575 generic.go:356] "Generic (PLEG): container finished" podID="92da4b99-1501-4784-9904-ac3dc444bf25" containerID="ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5" exitCode=0 Jul 12 16:31:45.916041 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.915937 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:45.916041 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.915974 2575 scope.go:117] "RemoveContainer" containerID="ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5" Jul 12 16:31:45.918875 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.918776 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c898e98-3140-45fe-bc99-368e1b4885c9" containerID="4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af" exitCode=0 Jul 12 16:31:45.918985 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.918886 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:45.921526 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.921497 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" event={"ID":"3a386d73-e480-4fee-a740-19265d43ccb5","Type":"ContainerStarted","Data":"d3e2068026252c25de097ef04aad0b865ce24f11a9e81c5e0cc21a43edeaab60"} Jul 12 16:31:45.923616 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.923596 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" event={"ID":"3094a87b-2bcb-4719-9ecc-d6118a842f10","Type":"ContainerStarted","Data":"bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3"} Jul 12 16:31:45.923710 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.923645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" event={"ID":"3094a87b-2bcb-4719-9ecc-d6118a842f10","Type":"ContainerStarted","Data":"0dd1a35eb4221e0894453b214f028577573cf637c2c16f1e5e5ba9ac4e23cd2e"} Jul 12 16:31:45.923761 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.923743 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:45.926689 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.926673 2575 scope.go:117] "RemoveContainer" containerID="ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5" Jul 12 16:31:45.926983 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:45.926964 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5\": container with ID starting with ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5 not found: ID does not exist" containerID="ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5" Jul 12 16:31:45.927024 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.926993 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5"} err="failed to get container status \"ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5\": rpc error: code = NotFound desc = could not find container \"ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5\": container with ID starting with ec0cdee1531cdd678a59272dda5e9964400539432a76015d515b17a911a9c8e5 not found: ID does not exist" Jul 12 16:31:45.927024 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.927009 2575 scope.go:117] "RemoveContainer" containerID="4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af" Jul 12 16:31:45.934973 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.934958 2575 scope.go:117] "RemoveContainer" containerID="4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af" Jul 12 16:31:45.935277 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:45.935250 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af\": container with ID starting with 4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af not found: ID does not exist" containerID="4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af" Jul 12 16:31:45.935325 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:45.935288 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af"} err="failed to get container status \"4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af\": rpc error: code = NotFound desc = could not find container \"4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af\": container with ID starting with 4c27e07a6ddb59d9db360fd5d0571665ea1e6b31842ff7e4b993719a72f2a5af not found: ID does not exist" Jul 12 16:31:46.033362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:46.033304 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" podStartSLOduration=13.033287228 podStartE2EDuration="13.033287228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:46.032920263 +0000 UTC m=+404.802348182" watchObservedRunningTime="2026-07-12 16:31:46.033287228 +0000 UTC m=+404.802715101" Jul 12 16:31:46.054485 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:46.054426 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k44fs" podStartSLOduration=2.613212933 podStartE2EDuration="24.054409323s" totalImagesPullingTime="21.44119639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:46.052464718 +0000 UTC m=+404.821892647" watchObservedRunningTime="2026-07-12 16:31:46.054409323 +0000 UTC m=+404.823837219" Jul 12 16:31:46.083400 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:46.083350 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" podStartSLOduration=13.083333172 podStartE2EDuration="13.083333172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:46.081107482 +0000 UTC m=+404.850535376" watchObservedRunningTime="2026-07-12 16:31:46.083333172 +0000 UTC m=+404.852761087" Jul 12 16:31:46.083400 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:46.083388 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ktxw2\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="92da4b99-1501-4784-9904-ac3dc444bf25" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ktxw2" Jul 12 16:31:46.085299 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:46.085276 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-84cx4\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="9c898e98-3140-45fe-bc99-368e1b4885c9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84cx4" Jul 12 16:31:46.109211 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:46.109122 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" podStartSLOduration=13.109099106 podStartE2EDuration="13.109099106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:46.105238412 +0000 UTC m=+404.874666330" watchObservedRunningTime="2026-07-12 16:31:46.109099106 +0000 UTC m=+404.878527003" Jul 12 16:31:54.096154 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:54.096119 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:54.196537 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:54.196504 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:55.889767 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:55.889709 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55dc84b75c-2tbh6" podUID="7203e0bf-5a0f-45ba-b76e-e006127f91da" containerName="console" containerID="cri-o://201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac" gracePeriod=15 Jul 12 16:31:56.147053 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.146984 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:31:56.260364 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260333 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c264m\" (UniqueName: \"kubernetes.io/projected/7203e0bf-5a0f-45ba-b76e-e006127f91da-kube-api-access-c264m\") pod \"7203e0bf-5a0f-45ba-b76e-e006127f91da\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " Jul 12 16:31:56.260557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260373 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-oauth-serving-cert\") pod \"7203e0bf-5a0f-45ba-b76e-e006127f91da\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " Jul 12 16:31:56.260557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260404 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-trusted-ca-bundle\") pod \"7203e0bf-5a0f-45ba-b76e-e006127f91da\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " Jul 12 16:31:56.260686 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260563 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-config\") pod \"7203e0bf-5a0f-45ba-b76e-e006127f91da\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " Jul 12 16:31:56.260686 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260622 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-oauth-config\") pod \"7203e0bf-5a0f-45ba-b76e-e006127f91da\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " Jul 12 16:31:56.260686 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260641 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-service-ca\") pod \"7203e0bf-5a0f-45ba-b76e-e006127f91da\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " Jul 12 16:31:56.260686 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260670 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-serving-cert\") pod \"7203e0bf-5a0f-45ba-b76e-e006127f91da\" (UID: \"7203e0bf-5a0f-45ba-b76e-e006127f91da\") " Jul 12 16:31:56.260882 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7203e0bf-5a0f-45ba-b76e-e006127f91da" (UID: "7203e0bf-5a0f-45ba-b76e-e006127f91da"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:31:56.260945 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260867 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7203e0bf-5a0f-45ba-b76e-e006127f91da" (UID: "7203e0bf-5a0f-45ba-b76e-e006127f91da"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:31:56.260998 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.260975 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-config" (OuterVolumeSpecName: "console-config") pod "7203e0bf-5a0f-45ba-b76e-e006127f91da" (UID: "7203e0bf-5a0f-45ba-b76e-e006127f91da"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:31:56.261054 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.261039 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-oauth-serving-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:56.261112 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.261056 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-service-ca" (OuterVolumeSpecName: "service-ca") pod "7203e0bf-5a0f-45ba-b76e-e006127f91da" (UID: "7203e0bf-5a0f-45ba-b76e-e006127f91da"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:31:56.261112 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.261060 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-trusted-ca-bundle\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:56.262606 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.262583 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7203e0bf-5a0f-45ba-b76e-e006127f91da-kube-api-access-c264m" (OuterVolumeSpecName: "kube-api-access-c264m") pod "7203e0bf-5a0f-45ba-b76e-e006127f91da" (UID: "7203e0bf-5a0f-45ba-b76e-e006127f91da"). InnerVolumeSpecName "kube-api-access-c264m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:31:56.262696 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.262637 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7203e0bf-5a0f-45ba-b76e-e006127f91da" (UID: "7203e0bf-5a0f-45ba-b76e-e006127f91da"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:31:56.262841 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.262821 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7203e0bf-5a0f-45ba-b76e-e006127f91da" (UID: "7203e0bf-5a0f-45ba-b76e-e006127f91da"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:31:56.361988 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.361944 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-oauth-config\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:56.361988 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.361980 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-service-ca\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:56.361988 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.361993 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-serving-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:56.362273 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.362005 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c264m\" (UniqueName: \"kubernetes.io/projected/7203e0bf-5a0f-45ba-b76e-e006127f91da-kube-api-access-c264m\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:56.362273 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.362017 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7203e0bf-5a0f-45ba-b76e-e006127f91da-console-config\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:56.536739 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.536704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:56.595664 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.595618 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:31:56.933258 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.933230 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:31:56.933725 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.933312 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wth6" Jul 12 16:31:56.933725 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.933360 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:56.987060 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.987023 2575 generic.go:356] "Generic (PLEG): container finished" podID="7203e0bf-5a0f-45ba-b76e-e006127f91da" containerID="201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac" exitCode=0 Jul 12 16:31:56.987251 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.987087 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dc84b75c-2tbh6" event={"ID":"7203e0bf-5a0f-45ba-b76e-e006127f91da","Type":"ContainerDied","Data":"201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac"} Jul 12 16:31:56.987251 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.987121 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55dc84b75c-2tbh6" event={"ID":"7203e0bf-5a0f-45ba-b76e-e006127f91da","Type":"ContainerDied","Data":"5035a837634f9e457c37c9905770dde5ef5892e959d57c529dcd3cb431423b0b"} Jul 12 16:31:56.987251 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.987141 2575 scope.go:117] "RemoveContainer" containerID="201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac" Jul 12 16:31:56.987251 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.987089 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55dc84b75c-2tbh6" Jul 12 16:31:56.997311 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.997259 2575 scope.go:117] "RemoveContainer" containerID="201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac" Jul 12 16:31:56.997581 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:56.997560 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac\": container with ID starting with 201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac not found: ID does not exist" containerID="201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac" Jul 12 16:31:56.997661 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:56.997594 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac"} err="failed to get container status \"201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac\": rpc error: code = NotFound desc = could not find container \"201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac\": container with ID starting with 201d8722d0eb212e87c862392cb379dbb629f2fd2adf43c109a660b6b34cddac not found: ID does not exist" Jul 12 16:31:57.013660 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.013628 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l"] Jul 12 16:31:57.013810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.013792 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" podUID="e5f861b5-1052-46fd-affe-31395aed358b" containerName="manager" containerID="cri-o://20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618" gracePeriod=10 Jul 12 16:31:57.022980 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.022940 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55dc84b75c-2tbh6"] Jul 12 16:31:57.029629 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.029603 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55dc84b75c-2tbh6"] Jul 12 16:31:57.265262 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.265227 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:57.340073 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.340036 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc"] Jul 12 16:31:57.340486 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.340469 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f861b5-1052-46fd-affe-31395aed358b" containerName="manager" Jul 12 16:31:57.340570 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.340488 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f861b5-1052-46fd-affe-31395aed358b" containerName="manager" Jul 12 16:31:57.340570 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.340536 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7203e0bf-5a0f-45ba-b76e-e006127f91da" containerName="console" Jul 12 16:31:57.340570 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.340547 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7203e0bf-5a0f-45ba-b76e-e006127f91da" containerName="console" Jul 12 16:31:57.340723 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.340629 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f861b5-1052-46fd-affe-31395aed358b" containerName="manager" Jul 12 16:31:57.340723 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.340644 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7203e0bf-5a0f-45ba-b76e-e006127f91da" containerName="console" Jul 12 16:31:57.343644 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.343624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:57.355642 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.355618 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc"] Jul 12 16:31:57.370562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.370537 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/e5f861b5-1052-46fd-affe-31395aed358b-kube-api-access-rdn65\") pod \"e5f861b5-1052-46fd-affe-31395aed358b\" (UID: \"e5f861b5-1052-46fd-affe-31395aed358b\") " Jul 12 16:31:57.370671 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.370654 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5f861b5-1052-46fd-affe-31395aed358b-extensions-socket-volume\") pod \"e5f861b5-1052-46fd-affe-31395aed358b\" (UID: \"e5f861b5-1052-46fd-affe-31395aed358b\") " Jul 12 16:31:57.371019 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.370998 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f861b5-1052-46fd-affe-31395aed358b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e5f861b5-1052-46fd-affe-31395aed358b" (UID: "e5f861b5-1052-46fd-affe-31395aed358b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:31:57.372532 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.372513 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f861b5-1052-46fd-affe-31395aed358b-kube-api-access-rdn65" (OuterVolumeSpecName: "kube-api-access-rdn65") pod "e5f861b5-1052-46fd-affe-31395aed358b" (UID: "e5f861b5-1052-46fd-affe-31395aed358b"). InnerVolumeSpecName "kube-api-access-rdn65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:31:57.471912 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.471875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qc9vc\" (UID: \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:57.472094 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.471961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6xk\" (UniqueName: \"kubernetes.io/projected/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-kube-api-access-nm6xk\") pod \"kuadrant-operator-controller-manager-86878c769f-qc9vc\" (UID: \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:57.472094 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.472045 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5f861b5-1052-46fd-affe-31395aed358b-extensions-socket-volume\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:57.472094 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.472070 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/e5f861b5-1052-46fd-affe-31395aed358b-kube-api-access-rdn65\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:31:57.573126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.573092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6xk\" (UniqueName: \"kubernetes.io/projected/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-kube-api-access-nm6xk\") pod \"kuadrant-operator-controller-manager-86878c769f-qc9vc\" (UID: \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:57.573302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.573153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qc9vc\" (UID: \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:57.573507 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.573489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qc9vc\" (UID: \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:57.585394 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.585363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6xk\" (UniqueName: \"kubernetes.io/projected/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-kube-api-access-nm6xk\") pod \"kuadrant-operator-controller-manager-86878c769f-qc9vc\" (UID: \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:57.653371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.653334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:57.808477 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.808455 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc"] Jul 12 16:31:57.810660 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:31:57.810636 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7632ebc1_863f_4e20_9ebb_7fd44e3b3258.slice/crio-1e7177596ea646bbafe7a527631b494554482220e53c695c3d247b0885796b29 WatchSource:0}: Error finding container 1e7177596ea646bbafe7a527631b494554482220e53c695c3d247b0885796b29: Status 404 returned error can't find the container with id 1e7177596ea646bbafe7a527631b494554482220e53c695c3d247b0885796b29 Jul 12 16:31:57.874806 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.874778 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7203e0bf-5a0f-45ba-b76e-e006127f91da" path="/var/lib/kubelet/pods/7203e0bf-5a0f-45ba-b76e-e006127f91da/volumes" Jul 12 16:31:57.994464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.994429 2575 generic.go:356] "Generic (PLEG): container finished" podID="e5f861b5-1052-46fd-affe-31395aed358b" containerID="20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618" exitCode=0 Jul 12 16:31:57.994927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.994481 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" Jul 12 16:31:57.994927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.994507 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" event={"ID":"e5f861b5-1052-46fd-affe-31395aed358b","Type":"ContainerDied","Data":"20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618"} Jul 12 16:31:57.994927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.994562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l" event={"ID":"e5f861b5-1052-46fd-affe-31395aed358b","Type":"ContainerDied","Data":"95379a3acd84aafc1c38d41fb0f4b4d78fd0953989cc73fafc5aabc5cbbecf07"} Jul 12 16:31:57.994927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.994595 2575 scope.go:117] "RemoveContainer" containerID="20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618" Jul 12 16:31:57.997315 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.997292 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" event={"ID":"7632ebc1-863f-4e20-9ebb-7fd44e3b3258","Type":"ContainerStarted","Data":"d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82"} Jul 12 16:31:57.997430 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.997335 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" event={"ID":"7632ebc1-863f-4e20-9ebb-7fd44e3b3258","Type":"ContainerStarted","Data":"1e7177596ea646bbafe7a527631b494554482220e53c695c3d247b0885796b29"} Jul 12 16:31:57.997430 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:57.997367 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:31:58.003619 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:58.003599 2575 scope.go:117] "RemoveContainer" containerID="20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618" Jul 12 16:31:58.003873 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:31:58.003855 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618\": container with ID starting with 20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618 not found: ID does not exist" containerID="20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618" Jul 12 16:31:58.003925 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:58.003881 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618"} err="failed to get container status \"20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618\": rpc error: code = NotFound desc = could not find container \"20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618\": container with ID starting with 20417c8c45a0f3148d39774655d910147dbb52deb729c968a3c2cbfde3c96618 not found: ID does not exist" Jul 12 16:31:58.022718 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:58.022653 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l"] Jul 12 16:31:58.026716 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:58.026691 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcv8l"] Jul 12 16:31:58.047858 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:58.047803 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" podStartSLOduration=1.047788904 podStartE2EDuration="1.047788904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:31:58.046512671 +0000 UTC m=+416.815940584" watchObservedRunningTime="2026-07-12 16:31:58.047788904 +0000 UTC m=+416.817216799" Jul 12 16:31:59.875103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:31:59.875073 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5f861b5-1052-46fd-affe-31395aed358b" path="/var/lib/kubelet/pods/e5f861b5-1052-46fd-affe-31395aed358b/volumes" Jul 12 16:32:00.913202 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:00.913160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:01.016743 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:01.016709 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:04.509323 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:04.509286 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:04.586011 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:04.585979 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:07.264898 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:07.264868 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:09.006657 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.006626 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:32:09.057477 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.057441 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt"] Jul 12 16:32:09.057731 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.057706 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" podUID="3094a87b-2bcb-4719-9ecc-d6118a842f10" containerName="manager" containerID="cri-o://bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3" gracePeriod=10 Jul 12 16:32:09.861982 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.861950 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:32:09.882763 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.882733 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3094a87b-2bcb-4719-9ecc-d6118a842f10-extensions-socket-volume\") pod \"3094a87b-2bcb-4719-9ecc-d6118a842f10\" (UID: \"3094a87b-2bcb-4719-9ecc-d6118a842f10\") " Jul 12 16:32:09.882924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.882839 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb4rv\" (UniqueName: \"kubernetes.io/projected/3094a87b-2bcb-4719-9ecc-d6118a842f10-kube-api-access-gb4rv\") pod \"3094a87b-2bcb-4719-9ecc-d6118a842f10\" (UID: \"3094a87b-2bcb-4719-9ecc-d6118a842f10\") " Jul 12 16:32:09.883157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.883125 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3094a87b-2bcb-4719-9ecc-d6118a842f10-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3094a87b-2bcb-4719-9ecc-d6118a842f10" (UID: "3094a87b-2bcb-4719-9ecc-d6118a842f10"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:32:09.885002 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.884978 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3094a87b-2bcb-4719-9ecc-d6118a842f10-kube-api-access-gb4rv" (OuterVolumeSpecName: "kube-api-access-gb4rv") pod "3094a87b-2bcb-4719-9ecc-d6118a842f10" (UID: "3094a87b-2bcb-4719-9ecc-d6118a842f10"). InnerVolumeSpecName "kube-api-access-gb4rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:32:09.983520 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.983437 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3094a87b-2bcb-4719-9ecc-d6118a842f10-extensions-socket-volume\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:32:09.983520 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:09.983465 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4rv\" (UniqueName: \"kubernetes.io/projected/3094a87b-2bcb-4719-9ecc-d6118a842f10-kube-api-access-gb4rv\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:32:10.074481 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.074446 2575 generic.go:356] "Generic (PLEG): container finished" podID="3094a87b-2bcb-4719-9ecc-d6118a842f10" containerID="bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3" exitCode=0 Jul 12 16:32:10.074924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.074495 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" Jul 12 16:32:10.074924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.074536 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" event={"ID":"3094a87b-2bcb-4719-9ecc-d6118a842f10","Type":"ContainerDied","Data":"bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3"} Jul 12 16:32:10.074924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.074605 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt" event={"ID":"3094a87b-2bcb-4719-9ecc-d6118a842f10","Type":"ContainerDied","Data":"0dd1a35eb4221e0894453b214f028577573cf637c2c16f1e5e5ba9ac4e23cd2e"} Jul 12 16:32:10.074924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.074643 2575 scope.go:117] "RemoveContainer" containerID="bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3" Jul 12 16:32:10.084117 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.084101 2575 scope.go:117] "RemoveContainer" containerID="bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3" Jul 12 16:32:10.084484 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:32:10.084463 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3\": container with ID starting with bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3 not found: ID does not exist" containerID="bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3" Jul 12 16:32:10.084571 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.084492 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3"} err="failed to get container status \"bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3\": rpc error: code = NotFound desc = could not find container \"bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3\": container with ID starting with bd6386afd72f7f6a11625c1ead4d91840fbb580538675f627ed12c088066e8f3 not found: ID does not exist" Jul 12 16:32:10.105516 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.105492 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt"] Jul 12 16:32:10.120789 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:10.120767 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7dnjt"] Jul 12 16:32:11.875749 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:11.875712 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3094a87b-2bcb-4719-9ecc-d6118a842f10" path="/var/lib/kubelet/pods/3094a87b-2bcb-4719-9ecc-d6118a842f10/volumes" Jul 12 16:32:13.126575 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.126540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:13.226903 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.226869 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:13.343954 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.343922 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst"] Jul 12 16:32:13.344380 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.344362 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3094a87b-2bcb-4719-9ecc-d6118a842f10" containerName="manager" Jul 12 16:32:13.344380 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.344382 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3094a87b-2bcb-4719-9ecc-d6118a842f10" containerName="manager" Jul 12 16:32:13.344583 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.344503 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3094a87b-2bcb-4719-9ecc-d6118a842f10" containerName="manager" Jul 12 16:32:13.370665 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.370638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:13.404680 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.404616 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst"] Jul 12 16:32:13.404791 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.404707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405540 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmrd\" (UniqueName: \"kubernetes.io/projected/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-kube-api-access-lkmrd\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.405810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.405802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.407536 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.407517 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-k25bg\"" Jul 12 16:32:13.506765 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.506730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmrd\" (UniqueName: \"kubernetes.io/projected/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-kube-api-access-lkmrd\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.506939 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.506768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.506939 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.506821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.506939 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.506871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.506939 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.506897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.507176 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.506940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.507176 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.506976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.507176 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.507001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.507176 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.507032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.507374 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.507271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.507859 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.507832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.514938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.508098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.514938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.508160 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.514938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.508360 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.514938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.514548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.515557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.515536 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.516281 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.516253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmrd\" (UniqueName: \"kubernetes.io/projected/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-kube-api-access-lkmrd\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.516399 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.516379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jlcst\" (UID: \"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.659474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.659397 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:13.714782 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.714750 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:13.929071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.929044 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst"] Jul 12 16:32:13.930703 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:32:13.930679 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4af49d_2c7c_4dbc_9e7a_4bd677a0a956.slice/crio-2b87cb9895e192b93847e34af1d0ec6c580079f84fcc2442ab8b985bfe1a3671 WatchSource:0}: Error finding container 2b87cb9895e192b93847e34af1d0ec6c580079f84fcc2442ab8b985bfe1a3671: Status 404 returned error can't find the container with id 2b87cb9895e192b93847e34af1d0ec6c580079f84fcc2442ab8b985bfe1a3671 Jul 12 16:32:13.932833 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.932797 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:32:13.932924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.932862 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:32:13.932924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:13.932892 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:32:14.099304 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:14.099263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" event={"ID":"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956","Type":"ContainerStarted","Data":"3ea7178846530544eb10f7b3de9b0706ca774c87305154c231f2fac597c253b9"} Jul 12 16:32:14.099451 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:14.099321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" event={"ID":"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956","Type":"ContainerStarted","Data":"2b87cb9895e192b93847e34af1d0ec6c580079f84fcc2442ab8b985bfe1a3671"} Jul 12 16:32:14.123816 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:14.123771 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" podStartSLOduration=1.123754593 podStartE2EDuration="1.123754593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:32:14.121998972 +0000 UTC m=+432.891426868" watchObservedRunningTime="2026-07-12 16:32:14.123754593 +0000 UTC m=+432.893182487" Jul 12 16:32:14.715129 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:14.715096 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:15.586296 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:15.586263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:15.658138 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:15.658110 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:15.720072 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:15.720040 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:16.110611 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:16.110583 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:16.111695 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:16.111672 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:32:31.577976 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.577874 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:31.685274 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.685226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:31.742659 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.742625 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:31.898223 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.898131 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:32:31.900671 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.900653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:31.903035 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.903010 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 12 16:32:31.911025 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.910993 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:32:31.937460 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.937425 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:32:31.961782 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.961738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpf7m\" (UniqueName: \"kubernetes.io/projected/88826bf9-5f86-4544-bed2-ea8f9aa135e9-kube-api-access-jpf7m\") pod \"limitador-limitador-54d6d988d5-jtvlq\" (UID: \"88826bf9-5f86-4544-bed2-ea8f9aa135e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:31.961991 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:31.961808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/88826bf9-5f86-4544-bed2-ea8f9aa135e9-config-file\") pod \"limitador-limitador-54d6d988d5-jtvlq\" (UID: \"88826bf9-5f86-4544-bed2-ea8f9aa135e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:32.062556 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.062520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpf7m\" (UniqueName: \"kubernetes.io/projected/88826bf9-5f86-4544-bed2-ea8f9aa135e9-kube-api-access-jpf7m\") pod \"limitador-limitador-54d6d988d5-jtvlq\" (UID: \"88826bf9-5f86-4544-bed2-ea8f9aa135e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:32.062744 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.062574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/88826bf9-5f86-4544-bed2-ea8f9aa135e9-config-file\") pod \"limitador-limitador-54d6d988d5-jtvlq\" (UID: \"88826bf9-5f86-4544-bed2-ea8f9aa135e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:32.063249 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.063228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/88826bf9-5f86-4544-bed2-ea8f9aa135e9-config-file\") pod \"limitador-limitador-54d6d988d5-jtvlq\" (UID: \"88826bf9-5f86-4544-bed2-ea8f9aa135e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:32.070443 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.070417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpf7m\" (UniqueName: \"kubernetes.io/projected/88826bf9-5f86-4544-bed2-ea8f9aa135e9-kube-api-access-jpf7m\") pod \"limitador-limitador-54d6d988d5-jtvlq\" (UID: \"88826bf9-5f86-4544-bed2-ea8f9aa135e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:32.212474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.212445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:32.345350 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.345320 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:32:32.346699 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:32:32.346673 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88826bf9_5f86_4544_bed2_ea8f9aa135e9.slice/crio-7df9e4dd760d91ca9125ae1d6cb07818bb923536006cfca418804f7a0723e07d WatchSource:0}: Error finding container 7df9e4dd760d91ca9125ae1d6cb07818bb923536006cfca418804f7a0723e07d: Status 404 returned error can't find the container with id 7df9e4dd760d91ca9125ae1d6cb07818bb923536006cfca418804f7a0723e07d Jul 12 16:32:32.359232 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.359210 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-6v8m9"] Jul 12 16:32:32.362036 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.362018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6v8m9" Jul 12 16:32:32.364286 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.364266 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ldwnj\"" Jul 12 16:32:32.368886 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.368867 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6v8m9"] Jul 12 16:32:32.390098 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.390078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:32.445607 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.445583 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:32.465897 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.465844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rtd\" (UniqueName: \"kubernetes.io/projected/3ee0a154-ca78-44b4-818a-3815ec4214f9-kube-api-access-74rtd\") pod \"authorino-58cd955675-6v8m9\" (UID: \"3ee0a154-ca78-44b4-818a-3815ec4214f9\") " pod="kuadrant-system/authorino-58cd955675-6v8m9" Jul 12 16:32:32.566842 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.566815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rtd\" (UniqueName: \"kubernetes.io/projected/3ee0a154-ca78-44b4-818a-3815ec4214f9-kube-api-access-74rtd\") pod \"authorino-58cd955675-6v8m9\" (UID: \"3ee0a154-ca78-44b4-818a-3815ec4214f9\") " pod="kuadrant-system/authorino-58cd955675-6v8m9" Jul 12 16:32:32.576783 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.576753 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rtd\" (UniqueName: \"kubernetes.io/projected/3ee0a154-ca78-44b4-818a-3815ec4214f9-kube-api-access-74rtd\") pod \"authorino-58cd955675-6v8m9\" (UID: \"3ee0a154-ca78-44b4-818a-3815ec4214f9\") " pod="kuadrant-system/authorino-58cd955675-6v8m9" Jul 12 16:32:32.671900 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.671869 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6v8m9" Jul 12 16:32:32.810325 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:32.810305 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6v8m9"] Jul 12 16:32:32.811688 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:32:32.811661 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ee0a154_ca78_44b4_818a_3815ec4214f9.slice/crio-c8a8b7e5de7542509738362233645ff285611d962b70d54e8184ede3fb8afc96 WatchSource:0}: Error finding container c8a8b7e5de7542509738362233645ff285611d962b70d54e8184ede3fb8afc96: Status 404 returned error can't find the container with id c8a8b7e5de7542509738362233645ff285611d962b70d54e8184ede3fb8afc96 Jul 12 16:32:33.209554 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:33.209519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" event={"ID":"88826bf9-5f86-4544-bed2-ea8f9aa135e9","Type":"ContainerStarted","Data":"7df9e4dd760d91ca9125ae1d6cb07818bb923536006cfca418804f7a0723e07d"} Jul 12 16:32:33.210885 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:33.210858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6v8m9" event={"ID":"3ee0a154-ca78-44b4-818a-3815ec4214f9","Type":"ContainerStarted","Data":"c8a8b7e5de7542509738362233645ff285611d962b70d54e8184ede3fb8afc96"} Jul 12 16:32:37.240764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:37.240720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6v8m9" event={"ID":"3ee0a154-ca78-44b4-818a-3815ec4214f9","Type":"ContainerStarted","Data":"f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d"} Jul 12 16:32:37.242585 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:37.242562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" event={"ID":"88826bf9-5f86-4544-bed2-ea8f9aa135e9","Type":"ContainerStarted","Data":"ea0b9dae48a633e0f443f833366f38b8051d6021b80c9b18cf68ee1558e414bf"} Jul 12 16:32:37.242716 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:37.242676 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:37.255301 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:37.255253 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-6v8m9" podStartSLOduration=1.292317355 podStartE2EDuration="5.255238076s" totalImagesPullingTime="3.962920721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:32:37.254611793 +0000 UTC m=+456.024039719" watchObservedRunningTime="2026-07-12 16:32:37.255238076 +0000 UTC m=+456.024665971" Jul 12 16:32:37.270563 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:37.270511 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" podStartSLOduration=1.84383992 podStartE2EDuration="6.270493479s" totalImagesPullingTime="4.426653559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:32:37.270340219 +0000 UTC m=+456.039768115" watchObservedRunningTime="2026-07-12 16:32:37.270493479 +0000 UTC m=+456.039921380" Jul 12 16:32:37.307048 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:37.307015 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6v8m9"] Jul 12 16:32:39.253320 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:39.253275 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-6v8m9" podUID="3ee0a154-ca78-44b4-818a-3815ec4214f9" containerName="authorino" containerID="cri-o://f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d" gracePeriod=30 Jul 12 16:32:39.498613 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:39.498586 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6v8m9" Jul 12 16:32:39.632459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:39.632368 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74rtd\" (UniqueName: \"kubernetes.io/projected/3ee0a154-ca78-44b4-818a-3815ec4214f9-kube-api-access-74rtd\") pod \"3ee0a154-ca78-44b4-818a-3815ec4214f9\" (UID: \"3ee0a154-ca78-44b4-818a-3815ec4214f9\") " Jul 12 16:32:39.634318 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:39.634292 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee0a154-ca78-44b4-818a-3815ec4214f9-kube-api-access-74rtd" (OuterVolumeSpecName: "kube-api-access-74rtd") pod "3ee0a154-ca78-44b4-818a-3815ec4214f9" (UID: "3ee0a154-ca78-44b4-818a-3815ec4214f9"). InnerVolumeSpecName "kube-api-access-74rtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:32:39.733176 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:39.733146 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74rtd\" (UniqueName: \"kubernetes.io/projected/3ee0a154-ca78-44b4-818a-3815ec4214f9-kube-api-access-74rtd\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:32:40.260237 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.260174 2575 generic.go:356] "Generic (PLEG): container finished" podID="3ee0a154-ca78-44b4-818a-3815ec4214f9" containerID="f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d" exitCode=0 Jul 12 16:32:40.260753 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.260241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6v8m9" Jul 12 16:32:40.260753 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.260259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6v8m9" event={"ID":"3ee0a154-ca78-44b4-818a-3815ec4214f9","Type":"ContainerDied","Data":"f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d"} Jul 12 16:32:40.260753 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.260315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6v8m9" event={"ID":"3ee0a154-ca78-44b4-818a-3815ec4214f9","Type":"ContainerDied","Data":"c8a8b7e5de7542509738362233645ff285611d962b70d54e8184ede3fb8afc96"} Jul 12 16:32:40.260753 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.260340 2575 scope.go:117] "RemoveContainer" containerID="f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d" Jul 12 16:32:40.269280 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.269266 2575 scope.go:117] "RemoveContainer" containerID="f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d" Jul 12 16:32:40.269540 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:32:40.269521 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d\": container with ID starting with f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d not found: ID does not exist" containerID="f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d" Jul 12 16:32:40.269585 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.269548 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d"} err="failed to get container status \"f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d\": rpc error: code = NotFound desc = could not find container \"f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d\": container with ID starting with f3038c95f6446ac65f2666219e8fad0aa138112721cde864cc0d6baaea11f07d not found: ID does not exist" Jul 12 16:32:40.276936 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.276910 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6v8m9"] Jul 12 16:32:40.279897 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:40.279874 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-6v8m9"] Jul 12 16:32:41.876312 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:41.876278 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ee0a154-ca78-44b4-818a-3815ec4214f9" path="/var/lib/kubelet/pods/3ee0a154-ca78-44b4-818a-3815ec4214f9/volumes" Jul 12 16:32:48.248877 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:48.248848 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jtvlq" Jul 12 16:32:51.631718 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:51.631677 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:51.732940 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:51.732903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:32:52.091337 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:32:52.091301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:03.702342 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.702305 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:03.800428 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.800391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:03.917033 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.916998 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-whppt"] Jul 12 16:33:03.917470 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.917452 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee0a154-ca78-44b4-818a-3815ec4214f9" containerName="authorino" Jul 12 16:33:03.917574 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.917472 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee0a154-ca78-44b4-818a-3815ec4214f9" containerName="authorino" Jul 12 16:33:03.917628 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.917587 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ee0a154-ca78-44b4-818a-3815ec4214f9" containerName="authorino" Jul 12 16:33:03.922041 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.922020 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-whppt" Jul 12 16:33:03.924561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.924539 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ldwnj\"" Jul 12 16:33:03.926491 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:03.926468 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-whppt"] Jul 12 16:33:04.040223 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.040107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htq6v\" (UniqueName: \"kubernetes.io/projected/497cd74e-804b-4a34-ab3a-93c5db428dd6-kube-api-access-htq6v\") pod \"authorino-5d4b67c446-whppt\" (UID: \"497cd74e-804b-4a34-ab3a-93c5db428dd6\") " pod="kuadrant-system/authorino-5d4b67c446-whppt" Jul 12 16:33:04.108148 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.108103 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-whppt"] Jul 12 16:33:04.108405 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:33:04.108385 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-htq6v], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-whppt" podUID="497cd74e-804b-4a34-ab3a-93c5db428dd6" Jul 12 16:33:04.133166 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.133140 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-vwnrx"] Jul 12 16:33:04.136437 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.136408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.138699 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.138682 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 12 16:33:04.140665 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.140649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htq6v\" (UniqueName: \"kubernetes.io/projected/497cd74e-804b-4a34-ab3a-93c5db428dd6-kube-api-access-htq6v\") pod \"authorino-5d4b67c446-whppt\" (UID: \"497cd74e-804b-4a34-ab3a-93c5db428dd6\") " pod="kuadrant-system/authorino-5d4b67c446-whppt" Jul 12 16:33:04.145853 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.145831 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-vwnrx"] Jul 12 16:33:04.148589 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.148558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htq6v\" (UniqueName: \"kubernetes.io/projected/497cd74e-804b-4a34-ab3a-93c5db428dd6-kube-api-access-htq6v\") pod \"authorino-5d4b67c446-whppt\" (UID: \"497cd74e-804b-4a34-ab3a-93c5db428dd6\") " pod="kuadrant-system/authorino-5d4b67c446-whppt" Jul 12 16:33:04.164011 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.163986 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-vwnrx"] Jul 12 16:33:04.164281 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:33:04.164253 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tvpnz tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-vwnrx" podUID="5c10917b-c2af-45c5-a31b-2caf35b6407f" Jul 12 16:33:04.192275 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.192243 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-755795685b-rzlcc"] Jul 12 16:33:04.195844 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.195826 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:33:04.202863 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.202839 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-755795685b-rzlcc"] Jul 12 16:33:04.241995 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.241962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c10917b-c2af-45c5-a31b-2caf35b6407f-tls-cert\") pod \"authorino-76877994-vwnrx\" (UID: \"5c10917b-c2af-45c5-a31b-2caf35b6407f\") " pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.242161 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.242011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvpnz\" (UniqueName: \"kubernetes.io/projected/5c10917b-c2af-45c5-a31b-2caf35b6407f-kube-api-access-tvpnz\") pod \"authorino-76877994-vwnrx\" (UID: \"5c10917b-c2af-45c5-a31b-2caf35b6407f\") " pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.343431 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.343341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c10917b-c2af-45c5-a31b-2caf35b6407f-tls-cert\") pod \"authorino-76877994-vwnrx\" (UID: \"5c10917b-c2af-45c5-a31b-2caf35b6407f\") " pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.343431 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.343394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvpnz\" (UniqueName: \"kubernetes.io/projected/5c10917b-c2af-45c5-a31b-2caf35b6407f-kube-api-access-tvpnz\") pod \"authorino-76877994-vwnrx\" (UID: \"5c10917b-c2af-45c5-a31b-2caf35b6407f\") " pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.343660 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.343449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhgt\" (UniqueName: \"kubernetes.io/projected/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-kube-api-access-nbhgt\") pod \"authorino-755795685b-rzlcc\" (UID: \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\") " pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:33:04.343660 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.343519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-tls-cert\") pod \"authorino-755795685b-rzlcc\" (UID: \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\") " pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:33:04.346010 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.345977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c10917b-c2af-45c5-a31b-2caf35b6407f-tls-cert\") pod \"authorino-76877994-vwnrx\" (UID: \"5c10917b-c2af-45c5-a31b-2caf35b6407f\") " pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.351557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.351532 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvpnz\" (UniqueName: \"kubernetes.io/projected/5c10917b-c2af-45c5-a31b-2caf35b6407f-kube-api-access-tvpnz\") pod \"authorino-76877994-vwnrx\" (UID: \"5c10917b-c2af-45c5-a31b-2caf35b6407f\") " pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.395413 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.395381 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.395573 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.395383 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-whppt" Jul 12 16:33:04.402625 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.402604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:04.405701 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.405681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-whppt" Jul 12 16:33:04.444550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.444514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhgt\" (UniqueName: \"kubernetes.io/projected/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-kube-api-access-nbhgt\") pod \"authorino-755795685b-rzlcc\" (UID: \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\") " pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:33:04.444725 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.444583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-tls-cert\") pod \"authorino-755795685b-rzlcc\" (UID: \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\") " pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:33:04.446909 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.446890 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-tls-cert\") pod \"authorino-755795685b-rzlcc\" (UID: \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\") " pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:33:04.451927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.451910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhgt\" (UniqueName: \"kubernetes.io/projected/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-kube-api-access-nbhgt\") pod \"authorino-755795685b-rzlcc\" (UID: \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\") " pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:33:04.507942 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.507907 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:33:04.545120 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.545090 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c10917b-c2af-45c5-a31b-2caf35b6407f-tls-cert\") pod \"5c10917b-c2af-45c5-a31b-2caf35b6407f\" (UID: \"5c10917b-c2af-45c5-a31b-2caf35b6407f\") " Jul 12 16:33:04.545120 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.545125 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvpnz\" (UniqueName: \"kubernetes.io/projected/5c10917b-c2af-45c5-a31b-2caf35b6407f-kube-api-access-tvpnz\") pod \"5c10917b-c2af-45c5-a31b-2caf35b6407f\" (UID: \"5c10917b-c2af-45c5-a31b-2caf35b6407f\") " Jul 12 16:33:04.545385 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.545150 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htq6v\" (UniqueName: \"kubernetes.io/projected/497cd74e-804b-4a34-ab3a-93c5db428dd6-kube-api-access-htq6v\") pod \"497cd74e-804b-4a34-ab3a-93c5db428dd6\" (UID: \"497cd74e-804b-4a34-ab3a-93c5db428dd6\") " Jul 12 16:33:04.547234 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.547206 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c10917b-c2af-45c5-a31b-2caf35b6407f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5c10917b-c2af-45c5-a31b-2caf35b6407f" (UID: "5c10917b-c2af-45c5-a31b-2caf35b6407f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:33:04.547366 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.547279 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497cd74e-804b-4a34-ab3a-93c5db428dd6-kube-api-access-htq6v" (OuterVolumeSpecName: "kube-api-access-htq6v") pod "497cd74e-804b-4a34-ab3a-93c5db428dd6" (UID: "497cd74e-804b-4a34-ab3a-93c5db428dd6"). InnerVolumeSpecName "kube-api-access-htq6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:33:04.547366 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.547306 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c10917b-c2af-45c5-a31b-2caf35b6407f-kube-api-access-tvpnz" (OuterVolumeSpecName: "kube-api-access-tvpnz") pod "5c10917b-c2af-45c5-a31b-2caf35b6407f" (UID: "5c10917b-c2af-45c5-a31b-2caf35b6407f"). InnerVolumeSpecName "kube-api-access-tvpnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:33:04.642374 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.642347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-755795685b-rzlcc"] Jul 12 16:33:04.644391 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:33:04.644370 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35c8d0fa_6a53_49b7_bc1f_65f0e14288eb.slice/crio-18dde71cb628c410b805687af4f88ecd849726ba3ef38e966747fe22b07e0af4 WatchSource:0}: Error finding container 18dde71cb628c410b805687af4f88ecd849726ba3ef38e966747fe22b07e0af4: Status 404 returned error can't find the container with id 18dde71cb628c410b805687af4f88ecd849726ba3ef38e966747fe22b07e0af4 Jul 12 16:33:04.645743 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.645722 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c10917b-c2af-45c5-a31b-2caf35b6407f-tls-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:33:04.645743 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.645743 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvpnz\" (UniqueName: \"kubernetes.io/projected/5c10917b-c2af-45c5-a31b-2caf35b6407f-kube-api-access-tvpnz\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:33:04.645882 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.645755 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htq6v\" (UniqueName: \"kubernetes.io/projected/497cd74e-804b-4a34-ab3a-93c5db428dd6-kube-api-access-htq6v\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:33:04.647802 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:04.647723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:05.404596 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.404512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-whppt" Jul 12 16:33:05.404973 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.404512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755795685b-rzlcc" event={"ID":"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb","Type":"ContainerStarted","Data":"f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381"} Jul 12 16:33:05.404973 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.404638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755795685b-rzlcc" event={"ID":"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb","Type":"ContainerStarted","Data":"18dde71cb628c410b805687af4f88ecd849726ba3ef38e966747fe22b07e0af4"} Jul 12 16:33:05.404973 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.404711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-vwnrx" Jul 12 16:33:05.424419 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.424383 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-755795685b-rzlcc" podStartSLOduration=1.054484736 podStartE2EDuration="1.424370305s" totalImagesPullingTime="369.885569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:33:05.423607627 +0000 UTC m=+484.193035540" watchObservedRunningTime="2026-07-12 16:33:05.424370305 +0000 UTC m=+484.193798200" Jul 12 16:33:05.448393 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.448366 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-whppt"] Jul 12 16:33:05.454834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.454807 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-whppt"] Jul 12 16:33:05.477382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.477352 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-vwnrx"] Jul 12 16:33:05.480455 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.480431 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-vwnrx"] Jul 12 16:33:05.876160 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.876127 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="497cd74e-804b-4a34-ab3a-93c5db428dd6" path="/var/lib/kubelet/pods/497cd74e-804b-4a34-ab3a-93c5db428dd6/volumes" Jul 12 16:33:05.876433 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:05.876420 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c10917b-c2af-45c5-a31b-2caf35b6407f" path="/var/lib/kubelet/pods/5c10917b-c2af-45c5-a31b-2caf35b6407f/volumes" Jul 12 16:33:09.726005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:09.725971 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:09.818286 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:09.818248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:09.893070 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:09.893038 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:13.899040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:13.899009 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:13.994099 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:13.994052 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:48.665927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:48.665885 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:49.077333 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:49.077301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:49.215374 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:49.215347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:49.520510 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:49.520467 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:49.624550 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:49.624519 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:49.752201 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:49.752151 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:33:57.686838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:57.686804 2575 generic.go:356] "Generic (PLEG): container finished" podID="5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956" containerID="3ea7178846530544eb10f7b3de9b0706ca774c87305154c231f2fac597c253b9" exitCode=0 Jul 12 16:33:57.687293 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:57.686857 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" event={"ID":"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956","Type":"ContainerDied","Data":"3ea7178846530544eb10f7b3de9b0706ca774c87305154c231f2fac597c253b9"} Jul 12 16:33:57.687293 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:57.687258 2575 scope.go:117] "RemoveContainer" containerID="3ea7178846530544eb10f7b3de9b0706ca774c87305154c231f2fac597c253b9" Jul 12 16:33:57.688211 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:57.688155 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:33:57.688320 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:57.688269 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:33:57.688389 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:57.688318 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:33:58.609045 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:58.609011 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:58.694244 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:58.694183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" event={"ID":"5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956","Type":"ContainerStarted","Data":"4fd9f4b3a6645c415a656f2e8f495b2549ac930882a9b1fb0bae0d8e4a810290"} Jul 12 16:33:58.709451 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:58.709412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:33:58.715648 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:58.715553 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:33:58.716830 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:58.716789 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jlcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 12 16:33:58.716974 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:58.716858 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" podUID="5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 12 16:33:59.716144 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:59.716091 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jlcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 12 16:33:59.716672 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:33:59.716168 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" podUID="5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 12 16:34:00.715777 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:00.715737 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jlcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 12 16:34:00.715981 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:00.715819 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" podUID="5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 12 16:34:01.111254 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:01.111136 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:34:01.721215 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:01.715397 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jlcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 12 16:34:01.721215 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:01.715455 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" podUID="5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 12 16:34:02.719651 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:02.719618 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:34:02.720647 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:02.720628 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jlcst" Jul 12 16:34:08.428882 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:08.428824 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:34:08.886753 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:08.886668 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:34:19.164780 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.164735 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx"] Jul 12 16:34:19.168447 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.168426 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.171321 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.171298 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 12 16:34:19.172481 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.172458 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dt8gh\"" Jul 12 16:34:19.172481 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.172477 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 12 16:34:19.172650 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.172477 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 12 16:34:19.179832 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.179810 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx"] Jul 12 16:34:19.194288 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.194254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:34:19.203939 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.203914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe93d17-d8e0-4441-897b-d56f51a8e346-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.204074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.203979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7jv\" (UniqueName: \"kubernetes.io/projected/dfe93d17-d8e0-4441-897b-d56f51a8e346-kube-api-access-kn7jv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.204074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.204011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.204074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.204028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.204074 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.204043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.204305 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.204129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305084 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7jv\" (UniqueName: \"kubernetes.io/projected/dfe93d17-d8e0-4441-897b-d56f51a8e346-kube-api-access-kn7jv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305084 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305369 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305369 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305369 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305369 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe93d17-d8e0-4441-897b-d56f51a8e346-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305577 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305537 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305577 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305552 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.305675 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.305603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.307666 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.307637 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfe93d17-d8e0-4441-897b-d56f51a8e346-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.307790 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.307732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe93d17-d8e0-4441-897b-d56f51a8e346-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.314020 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.313991 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7jv\" (UniqueName: \"kubernetes.io/projected/dfe93d17-d8e0-4441-897b-d56f51a8e346-kube-api-access-kn7jv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-klpgx\" (UID: \"dfe93d17-d8e0-4441-897b-d56f51a8e346\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.479408 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.479372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:19.618427 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.618397 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx"] Jul 12 16:34:19.619360 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:34:19.619330 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe93d17_d8e0_4441_897b_d56f51a8e346.slice/crio-b00686af80522ba14e464d6438b0cc5baef541d370e69803c06c87d0dac60dd5 WatchSource:0}: Error finding container b00686af80522ba14e464d6438b0cc5baef541d370e69803c06c87d0dac60dd5: Status 404 returned error can't find the container with id b00686af80522ba14e464d6438b0cc5baef541d370e69803c06c87d0dac60dd5 Jul 12 16:34:19.813620 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:19.813543 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" event={"ID":"dfe93d17-d8e0-4441-897b-d56f51a8e346","Type":"ContainerStarted","Data":"b00686af80522ba14e464d6438b0cc5baef541d370e69803c06c87d0dac60dd5"} Jul 12 16:34:20.077160 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:20.077059 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:34:21.828823 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:21.828789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" event={"ID":"dfe93d17-d8e0-4441-897b-d56f51a8e346","Type":"ContainerStarted","Data":"925ab3955ccb1fee74214c95c7bfbc46599ff3ebef806298c22bec44730016d5"} Jul 12 16:34:21.847254 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:21.847204 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" podStartSLOduration=1.723311477 podStartE2EDuration="2.847175601s" totalImagesPullingTime="1.123864124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:34:21.844491883 +0000 UTC m=+560.613919778" watchObservedRunningTime="2026-07-12 16:34:21.847175601 +0000 UTC m=+560.616603528" Jul 12 16:34:29.479806 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:29.479768 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:29.480261 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:29.479847 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:29.492397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:29.492375 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:29.882207 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:29.882112 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-klpgx" Jul 12 16:34:31.561645 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:31.561617 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:34:32.973938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:32.973906 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:34:43.369582 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.369545 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526"] Jul 12 16:34:43.373130 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.373110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.375555 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.375536 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 12 16:34:43.383327 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.383308 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526"] Jul 12 16:34:43.469367 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.469336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:34:43.521322 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.521281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2s8\" (UniqueName: \"kubernetes.io/projected/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-kube-api-access-bm2s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.521516 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.521325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.521516 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.521373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.521516 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.521476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.521681 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.521534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.521681 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.521568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623042 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.622962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623042 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.623009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623042 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.623035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623360 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.623078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2s8\" (UniqueName: \"kubernetes.io/projected/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-kube-api-access-bm2s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623360 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.623103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623360 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.623126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623540 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.623508 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623599 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.623546 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.623599 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.623585 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.625209 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.625173 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.625607 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.625587 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.630519 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.630492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2s8\" (UniqueName: \"kubernetes.io/projected/56464e90-ccb8-43bf-b58d-e2cf7b96cb6a-kube-api-access-bm2s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526\" (UID: \"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.684589 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.684559 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:43.821491 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.821468 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526"] Jul 12 16:34:43.822988 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:34:43.822964 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56464e90_ccb8_43bf_b58d_e2cf7b96cb6a.slice/crio-dbdb42a17eca48cea2ea6b33b2daa826f94121ca0045286d8f6735e407b654d0 WatchSource:0}: Error finding container dbdb42a17eca48cea2ea6b33b2daa826f94121ca0045286d8f6735e407b654d0: Status 404 returned error can't find the container with id dbdb42a17eca48cea2ea6b33b2daa826f94121ca0045286d8f6735e407b654d0 Jul 12 16:34:43.944289 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:43.944248 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" event={"ID":"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a","Type":"ContainerStarted","Data":"dbdb42a17eca48cea2ea6b33b2daa826f94121ca0045286d8f6735e407b654d0"} Jul 12 16:34:44.951729 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:44.951678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" event={"ID":"56464e90-ccb8-43bf-b58d-e2cf7b96cb6a","Type":"ContainerStarted","Data":"713e1d2bbdce842b472a2dd826540c0133369e815adba302114013b47bcea14c"} Jul 12 16:34:44.970335 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:44.970279 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" podStartSLOduration=1.7960463519999998 podStartE2EDuration="1.970260531s" totalImagesPullingTime="174.214179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:34:44.968031391 +0000 UTC m=+583.737459276" watchObservedRunningTime="2026-07-12 16:34:44.970260531 +0000 UTC m=+583.739688404" Jul 12 16:34:45.066040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:45.066003 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:34:53.684832 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:53.684796 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:53.685344 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:53.684879 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:53.697292 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:53.697270 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:54.017218 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:54.017114 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526" Jul 12 16:34:57.385875 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:57.385845 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:34:59.789024 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:34:59.788992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:35:01.755574 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:01.755535 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:35:01.757521 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:01.757503 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:35:02.173900 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:02.173818 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:35:03.212557 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:03.212511 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:35:03.313203 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:03.313155 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:35:10.595000 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:10.594964 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:35:10.696170 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:10.696132 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:35:12.464414 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.464380 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk"] Jul 12 16:35:12.468038 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.468018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.470468 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.470450 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 12 16:35:12.476511 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.476489 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk"] Jul 12 16:35:12.488161 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.488134 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:35:12.583362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.583326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.583546 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.583368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.583546 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.583445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.583546 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.583506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.583546 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.583526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjw67\" (UniqueName: \"kubernetes.io/projected/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-kube-api-access-qjw67\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.583729 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.583572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684071 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684072 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjw67\" (UniqueName: \"kubernetes.io/projected/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-kube-api-access-qjw67\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684364 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684364 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684364 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684364 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684599 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684691 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.684691 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.684661 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.686316 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.686295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.686470 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.686452 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.692125 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.692105 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjw67\" (UniqueName: \"kubernetes.io/projected/4291ea18-6d1a-4ab2-b67c-24f70f6b0044-kube-api-access-qjw67\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk\" (UID: \"4291ea18-6d1a-4ab2-b67c-24f70f6b0044\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.777492 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.777406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:12.916564 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:12.916529 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk"] Jul 12 16:35:12.918178 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:35:12.918146 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4291ea18_6d1a_4ab2_b67c_24f70f6b0044.slice/crio-af7a89f62f20bcd4f82b31d98320b4299af25b71c72c790d0911e90f9dc3babd WatchSource:0}: Error finding container af7a89f62f20bcd4f82b31d98320b4299af25b71c72c790d0911e90f9dc3babd: Status 404 returned error can't find the container with id af7a89f62f20bcd4f82b31d98320b4299af25b71c72c790d0911e90f9dc3babd Jul 12 16:35:13.105821 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:13.105793 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" event={"ID":"4291ea18-6d1a-4ab2-b67c-24f70f6b0044","Type":"ContainerStarted","Data":"af7a89f62f20bcd4f82b31d98320b4299af25b71c72c790d0911e90f9dc3babd"} Jul 12 16:35:14.112773 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:14.112731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" event={"ID":"4291ea18-6d1a-4ab2-b67c-24f70f6b0044","Type":"ContainerStarted","Data":"495133e573f6de9b2ac0af8c7db317a433ce855ce45f8b1176f3147c80ee9557"} Jul 12 16:35:14.131922 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:14.131869 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" podStartSLOduration=1.9765327400000001 podStartE2EDuration="2.131848698s" totalImagesPullingTime="155.315958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:35:14.131630188 +0000 UTC m=+612.901058115" watchObservedRunningTime="2026-07-12 16:35:14.131848698 +0000 UTC m=+612.901276615" Jul 12 16:35:15.581431 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:15.581395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:35:22.777744 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:22.777700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:22.778229 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:22.777779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:22.790334 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:22.790315 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:35:23.175660 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:35:23.175581 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk" Jul 12 16:36:07.877369 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:07.877340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:36:07.975325 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:07.975290 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:36:15.085030 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:15.084983 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:36:15.183315 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:15.183286 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:36:16.214344 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.214302 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:36:16.365347 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.365304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:36:16.429317 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.429271 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-755795685b-rzlcc"] Jul 12 16:36:16.429598 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.429547 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-755795685b-rzlcc" podUID="35c8d0fa-6a53-49b7-bc1f-65f0e14288eb" containerName="authorino" containerID="cri-o://f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381" gracePeriod=30 Jul 12 16:36:16.692464 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.692439 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:36:16.889025 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.888926 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbhgt\" (UniqueName: \"kubernetes.io/projected/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-kube-api-access-nbhgt\") pod \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\" (UID: \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\") " Jul 12 16:36:16.889239 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.889032 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-tls-cert\") pod \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\" (UID: \"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb\") " Jul 12 16:36:16.891130 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.891102 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-kube-api-access-nbhgt" (OuterVolumeSpecName: "kube-api-access-nbhgt") pod "35c8d0fa-6a53-49b7-bc1f-65f0e14288eb" (UID: "35c8d0fa-6a53-49b7-bc1f-65f0e14288eb"). InnerVolumeSpecName "kube-api-access-nbhgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:36:16.899790 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.899765 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "35c8d0fa-6a53-49b7-bc1f-65f0e14288eb" (UID: "35c8d0fa-6a53-49b7-bc1f-65f0e14288eb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 16:36:16.990497 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.990458 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhgt\" (UniqueName: \"kubernetes.io/projected/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-kube-api-access-nbhgt\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:36:16.990497 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:16.990489 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb-tls-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:36:17.470463 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.470428 2575 generic.go:356] "Generic (PLEG): container finished" podID="35c8d0fa-6a53-49b7-bc1f-65f0e14288eb" containerID="f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381" exitCode=0 Jul 12 16:36:17.470890 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.470465 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755795685b-rzlcc" Jul 12 16:36:17.470890 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.470479 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755795685b-rzlcc" event={"ID":"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb","Type":"ContainerDied","Data":"f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381"} Jul 12 16:36:17.470890 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.470509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755795685b-rzlcc" event={"ID":"35c8d0fa-6a53-49b7-bc1f-65f0e14288eb","Type":"ContainerDied","Data":"18dde71cb628c410b805687af4f88ecd849726ba3ef38e966747fe22b07e0af4"} Jul 12 16:36:17.470890 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.470526 2575 scope.go:117] "RemoveContainer" containerID="f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381" Jul 12 16:36:17.479950 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.479933 2575 scope.go:117] "RemoveContainer" containerID="f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381" Jul 12 16:36:17.480222 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:36:17.480176 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381\": container with ID starting with f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381 not found: ID does not exist" containerID="f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381" Jul 12 16:36:17.480284 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.480231 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381"} err="failed to get container status \"f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381\": rpc error: code = NotFound desc = could not find container \"f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381\": container with ID starting with f6895438f35d76381185772d795f053e8e03ff7b58e237a2ff5356f7e4041381 not found: ID does not exist" Jul 12 16:36:17.490994 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.490971 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-755795685b-rzlcc"] Jul 12 16:36:17.495122 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.495102 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-755795685b-rzlcc"] Jul 12 16:36:17.875743 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:17.875661 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c8d0fa-6a53-49b7-bc1f-65f0e14288eb" path="/var/lib/kubelet/pods/35c8d0fa-6a53-49b7-bc1f-65f0e14288eb/volumes" Jul 12 16:36:44.372744 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:44.372706 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:36:53.858641 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:36:53.858556 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:37:04.470869 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:37:04.470834 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:37:13.768568 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:37:13.768530 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:37:19.755644 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:37:19.755610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:37:24.764843 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:37:24.764806 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:37:26.959957 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:37:26.959928 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:37:28.278642 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:37:28.278605 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:37:36.405088 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:37:36.405055 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:37:52.872343 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:37:52.872304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:38:04.070706 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:04.070672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:38:09.969622 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:09.969582 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:38:11.148899 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:11.148863 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:11.244555 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:11.244522 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:11.370908 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:11.370873 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:14.796413 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:14.796376 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:14.897568 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:14.897537 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:17.046720 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:17.046688 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:17.150941 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:17.150910 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:17.250350 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:17.250310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:17.442654 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:17.442619 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:20.760839 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:20.760803 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:38:23.064864 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:23.064795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:23.128175 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:23.128144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:38:40.665608 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:40.665573 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:38:50.063751 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:38:50.063717 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:39:00.265493 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:39:00.265462 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:39:05.856396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:39:05.856358 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:39:09.459283 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:39:09.459246 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:39:09.512062 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:39:09.512006 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:39:16.273294 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:39:16.273255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:39:25.865592 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:39:25.865555 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:39:38.364769 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:39:38.364729 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:39:48.263265 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:39:48.263224 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:40:01.793769 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:40:01.793736 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:40:01.795810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:40:01.795789 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:40:09.664663 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:40:09.664626 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:40:21.064927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:40:21.064892 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:40:44.973825 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:40:44.973791 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:40:55.566852 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:40:55.566818 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:40:55.587625 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:40:55.587597 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:40:56.714408 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:40:56.714370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:41:10.568371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:41:10.568330 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:41:22.068158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:41:22.068121 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:41:31.804291 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:41:31.804203 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:41:41.472292 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:41:41.472255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:41:50.764957 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:41:50.764918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:42:15.764357 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:15.764318 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc"] Jul 12 16:42:15.764974 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:15.764556 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" podUID="7632ebc1-863f-4e20-9ebb-7fd44e3b3258" containerName="manager" containerID="cri-o://d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82" gracePeriod=10 Jul 12 16:42:18.423886 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.423863 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:42:18.500840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.500740 2575 generic.go:356] "Generic (PLEG): container finished" podID="7632ebc1-863f-4e20-9ebb-7fd44e3b3258" containerID="d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82" exitCode=0 Jul 12 16:42:18.500840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.500782 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" Jul 12 16:42:18.501040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.500830 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" event={"ID":"7632ebc1-863f-4e20-9ebb-7fd44e3b3258","Type":"ContainerDied","Data":"d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82"} Jul 12 16:42:18.501040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.500884 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc" event={"ID":"7632ebc1-863f-4e20-9ebb-7fd44e3b3258","Type":"ContainerDied","Data":"1e7177596ea646bbafe7a527631b494554482220e53c695c3d247b0885796b29"} Jul 12 16:42:18.501040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.500907 2575 scope.go:117] "RemoveContainer" containerID="d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82" Jul 12 16:42:18.510862 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.510838 2575 scope.go:117] "RemoveContainer" containerID="d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82" Jul 12 16:42:18.511125 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:42:18.511105 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82\": container with ID starting with d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82 not found: ID does not exist" containerID="d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82" Jul 12 16:42:18.511228 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.511140 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82"} err="failed to get container status \"d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82\": rpc error: code = NotFound desc = could not find container \"d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82\": container with ID starting with d6d497280d9cda9c71ed23e2066d65b54c07a30858fb26e1df277ba35789ef82 not found: ID does not exist" Jul 12 16:42:18.579598 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.579562 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-extensions-socket-volume\") pod \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\" (UID: \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\") " Jul 12 16:42:18.579785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.579640 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm6xk\" (UniqueName: \"kubernetes.io/projected/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-kube-api-access-nm6xk\") pod \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\" (UID: \"7632ebc1-863f-4e20-9ebb-7fd44e3b3258\") " Jul 12 16:42:18.579918 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.579894 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7632ebc1-863f-4e20-9ebb-7fd44e3b3258" (UID: "7632ebc1-863f-4e20-9ebb-7fd44e3b3258"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:42:18.581720 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.581697 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-kube-api-access-nm6xk" (OuterVolumeSpecName: "kube-api-access-nm6xk") pod "7632ebc1-863f-4e20-9ebb-7fd44e3b3258" (UID: "7632ebc1-863f-4e20-9ebb-7fd44e3b3258"). InnerVolumeSpecName "kube-api-access-nm6xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:42:18.680656 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.680616 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-extensions-socket-volume\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:42:18.680656 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.680650 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm6xk\" (UniqueName: \"kubernetes.io/projected/7632ebc1-863f-4e20-9ebb-7fd44e3b3258-kube-api-access-nm6xk\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:42:18.826816 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.826787 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc"] Jul 12 16:42:18.830103 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:18.830080 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qc9vc"] Jul 12 16:42:19.875886 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:42:19.875848 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7632ebc1-863f-4e20-9ebb-7fd44e3b3258" path="/var/lib/kubelet/pods/7632ebc1-863f-4e20-9ebb-7fd44e3b3258/volumes" Jul 12 16:43:16.892163 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.892128 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8"] Jul 12 16:43:16.892591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.892530 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7632ebc1-863f-4e20-9ebb-7fd44e3b3258" containerName="manager" Jul 12 16:43:16.892591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.892542 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7632ebc1-863f-4e20-9ebb-7fd44e3b3258" containerName="manager" Jul 12 16:43:16.892591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.892573 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c8d0fa-6a53-49b7-bc1f-65f0e14288eb" containerName="authorino" Jul 12 16:43:16.892591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.892581 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c8d0fa-6a53-49b7-bc1f-65f0e14288eb" containerName="authorino" Jul 12 16:43:16.892722 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.892642 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7632ebc1-863f-4e20-9ebb-7fd44e3b3258" containerName="manager" Jul 12 16:43:16.892722 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.892656 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c8d0fa-6a53-49b7-bc1f-65f0e14288eb" containerName="authorino" Jul 12 16:43:16.896940 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.896923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:16.900881 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.900861 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rl62l\"" Jul 12 16:43:16.910028 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.910006 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8"] Jul 12 16:43:16.997949 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.997914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vmkh8\" (UID: \"5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:16.998136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:16.998018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxc87\" (UniqueName: \"kubernetes.io/projected/5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd-kube-api-access-pxc87\") pod \"kuadrant-operator-controller-manager-86878c769f-vmkh8\" (UID: \"5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:17.098856 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.098817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vmkh8\" (UID: \"5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:17.099043 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.098919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxc87\" (UniqueName: \"kubernetes.io/projected/5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd-kube-api-access-pxc87\") pod \"kuadrant-operator-controller-manager-86878c769f-vmkh8\" (UID: \"5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:17.099218 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.099179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vmkh8\" (UID: \"5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:17.107340 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.107319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxc87\" (UniqueName: \"kubernetes.io/projected/5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd-kube-api-access-pxc87\") pod \"kuadrant-operator-controller-manager-86878c769f-vmkh8\" (UID: \"5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:17.208352 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.208324 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:17.344452 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.344424 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8"] Jul 12 16:43:17.346127 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:43:17.346103 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2fcddd_cb72_47b9_a9ce_5cc660cb8dbd.slice/crio-dfa4bbd6333e1d0363172c8b8955e67c1f33c744c8f9f2b7e645ffd8f82c1d99 WatchSource:0}: Error finding container dfa4bbd6333e1d0363172c8b8955e67c1f33c744c8f9f2b7e645ffd8f82c1d99: Status 404 returned error can't find the container with id dfa4bbd6333e1d0363172c8b8955e67c1f33c744c8f9f2b7e645ffd8f82c1d99 Jul 12 16:43:17.834566 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.834526 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" event={"ID":"5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd","Type":"ContainerStarted","Data":"9a0fc12fa27cebbccfeea30d0c13e8980bb0adbfece4f3bfc343ce97262e3965"} Jul 12 16:43:17.834745 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.834591 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:17.834745 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.834614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" event={"ID":"5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd","Type":"ContainerStarted","Data":"dfa4bbd6333e1d0363172c8b8955e67c1f33c744c8f9f2b7e645ffd8f82c1d99"} Jul 12 16:43:17.854632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:17.854585 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" podStartSLOduration=1.8545690559999999 podStartE2EDuration="1.854569056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:43:17.851884036 +0000 UTC m=+1096.621311945" watchObservedRunningTime="2026-07-12 16:43:17.854569056 +0000 UTC m=+1096.623996951" Jul 12 16:43:28.842046 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:28.842018 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vmkh8" Jul 12 16:43:38.661002 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:38.660960 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:43:50.364292 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:50.364256 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:43:59.270120 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:43:59.270077 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:44:09.864274 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:44:09.864238 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:44:21.666647 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:44:21.666611 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:44:30.963914 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:44:30.963879 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:44:39.975702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:44:39.975665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:44:51.067062 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:44:51.067023 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:45:01.067498 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:01.067449 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:45:01.841090 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:01.841059 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:45:01.843858 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:01.843834 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:45:07.911072 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:07.911033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:45:07.995927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:07.995899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:45:10.370977 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:10.370940 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:45:21.559307 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:21.559271 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:45:43.572996 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:43.572956 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:45:53.264889 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:45:53.264805 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:46:03.167872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:46:03.167836 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:46:12.866342 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:46:12.866302 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:46:14.206839 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:46:14.206793 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:46:28.160535 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:46:28.160500 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:46:42.370347 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:46:42.370302 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:46:51.467279 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:46:51.467235 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:47:04.571167 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:47:04.571132 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:47:13.875032 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:47:13.874991 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:47:19.030578 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:47:19.030548 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:47:27.272313 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:47:27.272233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:47:36.865624 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:47:36.865589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:47:46.762041 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:47:46.761999 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:47:58.671630 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:47:58.671593 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:48:11.166657 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:48:11.166620 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:48:20.167293 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:48:20.167259 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:48:33.467493 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:48:33.467449 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:48:45.071588 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:48:45.071545 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:48:56.271101 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:48:56.271020 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:48:58.327080 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:48:58.327023 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:05.203715 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:05.203676 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:06.265481 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:06.265448 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:49:13.459545 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:13.459511 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:18.559533 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:18.559475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:21.487417 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:21.487375 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:21.590303 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:21.590235 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:21.925935 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:21.925910 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:22.970438 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:22.970400 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:49:27.612442 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.612406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:27.675729 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.675689 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:27.791042 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.791005 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9"] Jul 12 16:49:27.795364 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.795339 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.797951 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.797931 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-755d03bc-openshift-default-dockercfg-knchb\"" Jul 12 16:49:27.817995 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.817965 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9"] Jul 12 16:49:27.824733 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.824697 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:27.846733 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-credential-socket\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.846866 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08eb42e-7603-488b-b44d-92e0c260532c-istio-podinfo\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.846866 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-certs\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.846866 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-envoy\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.846866 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-data\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.847003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mctz\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-kube-api-access-2mctz\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.847003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-socket\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.847003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08eb42e-7603-488b-b44d-92e0c260532c-istiod-ca-cert\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.847003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.846957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-istio-token\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.925761 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.925668 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:27.947642 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-data\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.947756 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mctz\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-kube-api-access-2mctz\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.947756 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-socket\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.947756 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08eb42e-7603-488b-b44d-92e0c260532c-istiod-ca-cert\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.947756 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-istio-token\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.947955 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-credential-socket\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.947955 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08eb42e-7603-488b-b44d-92e0c260532c-istio-podinfo\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.947955 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-certs\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.947955 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.947850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-envoy\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.948166 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.948032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-data\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.948166 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.948047 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-socket\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.948315 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.948207 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-credential-socket\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.948315 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.948255 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-certs\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.948450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.948425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08eb42e-7603-488b-b44d-92e0c260532c-istiod-ca-cert\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.950174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.950154 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-envoy\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.950413 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.950394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08eb42e-7603-488b-b44d-92e0c260532c-istio-podinfo\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.955526 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.955502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mctz\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-kube-api-access-2mctz\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:27.955797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:27.955781 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-istio-token\") pod \"e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:28.106126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:28.106087 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:28.245088 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:28.245063 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9"] Jul 12 16:49:28.246387 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:49:28.246351 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08eb42e_7603_488b_b44d_92e0c260532c.slice/crio-63303d15cb55a5788a5025bb795c06e56efc5cdf52cc5548086bff358f87993a WatchSource:0}: Error finding container 63303d15cb55a5788a5025bb795c06e56efc5cdf52cc5548086bff358f87993a: Status 404 returned error can't find the container with id 63303d15cb55a5788a5025bb795c06e56efc5cdf52cc5548086bff358f87993a Jul 12 16:49:28.248568 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:28.248533 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:28.248670 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:28.248598 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:28.248670 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:28.248627 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:28.915037 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:28.914988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" event={"ID":"a08eb42e-7603-488b-b44d-92e0c260532c","Type":"ContainerStarted","Data":"7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627"} Jul 12 16:49:28.915523 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:28.915087 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" event={"ID":"a08eb42e-7603-488b-b44d-92e0c260532c","Type":"ContainerStarted","Data":"63303d15cb55a5788a5025bb795c06e56efc5cdf52cc5548086bff358f87993a"} Jul 12 16:49:28.935951 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:28.935896 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" podStartSLOduration=1.935862129 podStartE2EDuration="1.935862129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:49:28.933850566 +0000 UTC m=+1467.703278481" watchObservedRunningTime="2026-07-12 16:49:28.935862129 +0000 UTC m=+1467.705290024" Jul 12 16:49:29.008311 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.008245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:29.091146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.091116 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:29.107141 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.107114 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:29.112486 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.112462 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:29.153446 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.153399 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:29.324640 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.324610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:29.668083 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.667983 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:29.769075 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.769039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:29.920934 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.920848 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:29.921867 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:29.921848 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:31.458797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:31.458767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:33.865241 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:33.865183 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:49:34.128796 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:34.128715 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:34.306820 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:34.306779 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9"] Jul 12 16:49:34.307057 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:34.307019 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" podUID="a08eb42e-7603-488b-b44d-92e0c260532c" containerName="istio-proxy" containerID="cri-o://7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627" gracePeriod=30 Jul 12 16:49:34.334207 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:34.331693 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:34.427963 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:34.427920 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:34.527323 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:34.527290 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:37.551927 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:37.551879 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:39.555149 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.555125 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:39.658672 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658590 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08eb42e-7603-488b-b44d-92e0c260532c-istiod-ca-cert\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.658672 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658650 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mctz\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-kube-api-access-2mctz\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.658897 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658687 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-envoy\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.658897 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658778 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-credential-socket\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.658897 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658812 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-socket\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.658897 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658838 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-istio-token\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.658897 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658876 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-certs\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.659146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658902 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08eb42e-7603-488b-b44d-92e0c260532c-istio-podinfo\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.659146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658932 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-data\") pod \"a08eb42e-7603-488b-b44d-92e0c260532c\" (UID: \"a08eb42e-7603-488b-b44d-92e0c260532c\") " Jul 12 16:49:39.659146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.658970 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08eb42e-7603-488b-b44d-92e0c260532c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:49:39.659146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.659072 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:39.659146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.659083 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:39.659443 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.659327 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:39.659443 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.659382 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-credential-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.659443 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.659400 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.659443 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.659410 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08eb42e-7603-488b-b44d-92e0c260532c-istiod-ca-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.659834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.659813 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:39.660954 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.660924 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-kube-api-access-2mctz" (OuterVolumeSpecName: "kube-api-access-2mctz") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "kube-api-access-2mctz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:49:39.661079 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.661056 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a08eb42e-7603-488b-b44d-92e0c260532c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 16:49:39.661324 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.661304 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:39.661390 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.661370 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a08eb42e-7603-488b-b44d-92e0c260532c" (UID: "a08eb42e-7603-488b-b44d-92e0c260532c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:49:39.760840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.760806 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.760840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.760833 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-workload-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.760840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.760843 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08eb42e-7603-488b-b44d-92e0c260532c-istio-podinfo\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.761076 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.760851 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-data\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.761076 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.760860 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mctz\" (UniqueName: \"kubernetes.io/projected/a08eb42e-7603-488b-b44d-92e0c260532c-kube-api-access-2mctz\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.761076 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.760869 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08eb42e-7603-488b-b44d-92e0c260532c-istio-envoy\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:39.983648 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.983612 2575 generic.go:356] "Generic (PLEG): container finished" podID="a08eb42e-7603-488b-b44d-92e0c260532c" containerID="7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627" exitCode=0 Jul 12 16:49:39.983817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.983665 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" Jul 12 16:49:39.983817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.983700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" event={"ID":"a08eb42e-7603-488b-b44d-92e0c260532c","Type":"ContainerDied","Data":"7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627"} Jul 12 16:49:39.983817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.983778 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9" event={"ID":"a08eb42e-7603-488b-b44d-92e0c260532c","Type":"ContainerDied","Data":"63303d15cb55a5788a5025bb795c06e56efc5cdf52cc5548086bff358f87993a"} Jul 12 16:49:39.983817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.983801 2575 scope.go:117] "RemoveContainer" containerID="7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627" Jul 12 16:49:39.993949 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.993922 2575 scope.go:117] "RemoveContainer" containerID="7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627" Jul 12 16:49:39.994247 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:49:39.994229 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627\": container with ID starting with 7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627 not found: ID does not exist" containerID="7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627" Jul 12 16:49:39.994309 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:39.994255 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627"} err="failed to get container status \"7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627\": rpc error: code = NotFound desc = could not find container \"7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627\": container with ID starting with 7d4549370df992dff2b898d58fe4aa62791dcb946eeeab2b9bd616d7e87c8627 not found: ID does not exist" Jul 12 16:49:40.002753 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.002714 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9"] Jul 12 16:49:40.007199 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.007168 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-755d03bc-openshift-default-7cdbd97789-ltfz9"] Jul 12 16:49:40.144427 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.144396 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:40.207115 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.207079 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:40.313355 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.313142 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl"] Jul 12 16:49:40.313753 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.313729 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a08eb42e-7603-488b-b44d-92e0c260532c" containerName="istio-proxy" Jul 12 16:49:40.313912 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.313895 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a08eb42e-7603-488b-b44d-92e0c260532c" containerName="istio-proxy" Jul 12 16:49:40.314165 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.314149 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a08eb42e-7603-488b-b44d-92e0c260532c" containerName="istio-proxy" Jul 12 16:49:40.319484 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.319461 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.322255 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.322212 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-183966ff-openshift-default-dockercfg-zcgpp\"" Jul 12 16:49:40.329777 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.329755 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl"] Jul 12 16:49:40.355435 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.355405 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:40.455962 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.455925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:40.467064 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-data\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.467161 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-token\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.467161 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-envoy\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.467254 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-certs\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.467254 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwfv\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-kube-api-access-ndwfv\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.467254 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-podinfo\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.467353 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istiod-ca-cert\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.467353 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-socket\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.467414 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.467359 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-credential-socket\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.568587 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-socket\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.568587 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-credential-socket\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.568587 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-data\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-token\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-envoy\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-certs\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwfv\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-kube-api-access-ndwfv\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-podinfo\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istiod-ca-cert\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.568923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-socket\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.569003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-credential-socket\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569132 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.569069 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-data\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569532 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.569299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-certs\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.569532 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.569495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istiod-ca-cert\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.570901 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.570879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-envoy\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.571145 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.571127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-podinfo\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.576681 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.576660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-token\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.576681 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.576677 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwfv\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-kube-api-access-ndwfv\") pod \"e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.633165 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.633132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:40.773327 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.773304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl"] Jul 12 16:49:40.775291 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:49:40.775264 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08cf9fe1_fd64_46c2_a5d0_9dcdd1ef5061.slice/crio-6f3436144d61c6908c773ec6b0a5c9895b8b4f3271b9039d0830f7d9b334bcbf WatchSource:0}: Error finding container 6f3436144d61c6908c773ec6b0a5c9895b8b4f3271b9039d0830f7d9b334bcbf: Status 404 returned error can't find the container with id 6f3436144d61c6908c773ec6b0a5c9895b8b4f3271b9039d0830f7d9b334bcbf Jul 12 16:49:40.777502 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.777456 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:40.777573 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.777551 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:40.777615 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.777601 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:40.990748 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.990711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" event={"ID":"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061","Type":"ContainerStarted","Data":"19b8f294d744c48a8f64bc1426a1774de0ca5c25635f3288cfa577f0621ee616"} Jul 12 16:49:40.990936 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:40.990767 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" event={"ID":"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061","Type":"ContainerStarted","Data":"6f3436144d61c6908c773ec6b0a5c9895b8b4f3271b9039d0830f7d9b334bcbf"} Jul 12 16:49:41.013501 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.013439 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" podStartSLOduration=1.01341867 podStartE2EDuration="1.01341867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:49:41.010218618 +0000 UTC m=+1479.779646543" watchObservedRunningTime="2026-07-12 16:49:41.01341867 +0000 UTC m=+1479.782846566" Jul 12 16:49:41.440911 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.440871 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:41.541219 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.541157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:41.634001 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.633961 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:41.638903 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.638879 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:41.679533 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.679504 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:41.781141 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.781054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:41.875811 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.875779 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a08eb42e-7603-488b-b44d-92e0c260532c" path="/var/lib/kubelet/pods/a08eb42e-7603-488b-b44d-92e0c260532c/volumes" Jul 12 16:49:41.998277 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.998251 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:41.999113 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:41.999094 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:42.295084 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:42.295046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:42.393561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:42.393463 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:43.409012 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:43.408982 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:45.266954 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:45.266919 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:49:46.568413 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:46.568375 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:47.065538 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:47.065503 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:52.718767 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:52.718730 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl"] Jul 12 16:49:52.719267 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:52.718935 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" podUID="08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" containerName="istio-proxy" containerID="cri-o://19b8f294d744c48a8f64bc1426a1774de0ca5c25635f3288cfa577f0621ee616" gracePeriod=30 Jul 12 16:49:52.741676 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:52.741646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:52.841695 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:52.841652 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:53.084622 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.084592 2575 generic.go:356] "Generic (PLEG): container finished" podID="08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" containerID="19b8f294d744c48a8f64bc1426a1774de0ca5c25635f3288cfa577f0621ee616" exitCode=0 Jul 12 16:49:53.084817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.084672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" event={"ID":"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061","Type":"ContainerDied","Data":"19b8f294d744c48a8f64bc1426a1774de0ca5c25635f3288cfa577f0621ee616"} Jul 12 16:49:53.181687 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.181659 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:53.187229 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187206 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-envoy\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187235 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-podinfo\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187283 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-token\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187306 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-socket\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187324 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istiod-ca-cert\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187362 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187346 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndwfv\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-kube-api-access-ndwfv\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187652 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187389 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-credential-socket\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187652 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187422 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-data\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187652 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187446 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-certs\") pod \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\" (UID: \"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061\") " Jul 12 16:49:53.187652 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187599 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:53.187870 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187699 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:49:53.187870 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187761 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:53.187870 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187799 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:53.187870 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187804 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:53.187870 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.187817 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istiod-ca-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:53.188449 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.188421 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-data" (OuterVolumeSpecName: "istio-data") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:53.189728 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.189695 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-kube-api-access-ndwfv" (OuterVolumeSpecName: "kube-api-access-ndwfv") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "kube-api-access-ndwfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:49:53.189868 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.189845 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 16:49:53.189868 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.189858 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-token" (OuterVolumeSpecName: "istio-token") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:49:53.189960 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.189855 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" (UID: "08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:49:53.288319 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.288288 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:53.288319 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.288313 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndwfv\" (UniqueName: \"kubernetes.io/projected/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-kube-api-access-ndwfv\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:53.288319 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.288322 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-credential-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:53.288536 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.288332 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-data\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:53.288536 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.288341 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-workload-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:53.288536 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.288349 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-envoy\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:53.288536 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:53.288356 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061-istio-podinfo\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:49:54.091371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:54.091345 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" Jul 12 16:49:54.091371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:54.091354 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl" event={"ID":"08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061","Type":"ContainerDied","Data":"6f3436144d61c6908c773ec6b0a5c9895b8b4f3271b9039d0830f7d9b334bcbf"} Jul 12 16:49:54.091848 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:54.091411 2575 scope.go:117] "RemoveContainer" containerID="19b8f294d744c48a8f64bc1426a1774de0ca5c25635f3288cfa577f0621ee616" Jul 12 16:49:54.109171 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:54.109144 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl"] Jul 12 16:49:54.116100 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:54.116067 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-183966ff-openshift-default-654dd9b777-6dgvl"] Jul 12 16:49:55.875322 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:55.875285 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" path="/var/lib/kubelet/pods/08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061/volumes" Jul 12 16:49:57.061512 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:57.061482 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:57.168034 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:57.168004 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:49:58.511640 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.511588 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:58.566674 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.566641 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:58.687789 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.687756 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k"] Jul 12 16:49:58.688156 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.688142 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" containerName="istio-proxy" Jul 12 16:49:58.688156 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.688155 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" containerName="istio-proxy" Jul 12 16:49:58.688332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.688283 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="08cf9fe1-fd64-46c2-a5d0-9dcdd1ef5061" containerName="istio-proxy" Jul 12 16:49:58.692827 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.692803 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.695650 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.695623 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b44a9a33-openshift-default-dockercfg-lhjtg\"" Jul 12 16:49:58.704341 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.704319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k"] Jul 12 16:49:58.714114 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.714088 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:58.733369 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istiod-ca-cert\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.733488 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733375 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-credential-socket\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.733488 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-certs\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.733488 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-token\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.733598 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-data\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.733598 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-socket\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.733598 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhng\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-kube-api-access-bkhng\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.733598 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-envoy\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.733772 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.733605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-podinfo\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.788646 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.788545 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:58.834702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istiod-ca-cert\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.834838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-credential-socket\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.834838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-certs\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.834838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-token\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.834958 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-data\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.834958 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-socket\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.834958 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhng\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-kube-api-access-bkhng\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.835117 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-envoy\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.835117 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.834982 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-podinfo\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.835267 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.835244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-credential-socket\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.835336 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.835275 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-data\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.835394 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.835356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-certs\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.835488 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.835467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-socket\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.835861 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.835473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istiod-ca-cert\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.837379 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.837353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-envoy\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.837543 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.837526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-podinfo\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.842622 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.842600 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-token\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:58.842757 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:58.842737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhng\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-kube-api-access-bkhng\") pod \"e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:59.003835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:59.003793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:49:59.148790 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:59.148765 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k"] Jul 12 16:49:59.151543 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:49:59.151514 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915b4d84_1d93_44d7_8d3e_3c7e5708278d.slice/crio-8f5c570c22041554fea2610f12f071f72a0e6a79c439d1652ce1fe0284bafe0a WatchSource:0}: Error finding container 8f5c570c22041554fea2610f12f071f72a0e6a79c439d1652ce1fe0284bafe0a: Status 404 returned error can't find the container with id 8f5c570c22041554fea2610f12f071f72a0e6a79c439d1652ce1fe0284bafe0a Jul 12 16:49:59.153615 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:59.153580 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:59.153702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:59.153643 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:59.153702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:59.153673 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:49:59.900712 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:59.900677 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:49:59.990592 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:49:59.990559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:00.053256 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:00.053214 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:00.130980 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:00.130951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" event={"ID":"915b4d84-1d93-44d7-8d3e-3c7e5708278d","Type":"ContainerStarted","Data":"b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136"} Jul 12 16:50:00.131146 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:00.131001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" event={"ID":"915b4d84-1d93-44d7-8d3e-3c7e5708278d","Type":"ContainerStarted","Data":"8f5c570c22041554fea2610f12f071f72a0e6a79c439d1652ce1fe0284bafe0a"} Jul 12 16:50:00.166040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:00.165935 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" podStartSLOduration=2.16591455 podStartE2EDuration="2.16591455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:50:00.161790841 +0000 UTC m=+1498.931218756" watchObservedRunningTime="2026-07-12 16:50:00.16591455 +0000 UTC m=+1498.935342445" Jul 12 16:50:00.192785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:00.192749 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:00.593299 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:00.593261 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:00.695633 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:00.695596 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:01.004628 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:01.004593 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:50:01.010502 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:01.010474 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:50:01.137392 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:01.137365 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:50:01.138321 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:01.138301 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:50:01.882062 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:01.882037 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:50:01.885244 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:01.885222 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:50:02.179504 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:02.179469 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:04.259505 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:04.259469 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:04.431154 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:04.431110 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k"] Jul 12 16:50:04.431479 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:04.431448 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" podUID="915b4d84-1d93-44d7-8d3e-3c7e5708278d" containerName="istio-proxy" containerID="cri-o://b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136" gracePeriod=30 Jul 12 16:50:04.451883 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:04.451849 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:04.553764 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:04.553673 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:04.639201 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:04.639144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:06.062090 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:06.062051 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:09.687293 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.687269 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:50:09.735126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-socket\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735085 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-token\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735109 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkhng\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-kube-api-access-bkhng\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735407 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735141 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-certs\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735407 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735247 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-data\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735407 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735324 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-credential-socket\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735407 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735362 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istiod-ca-cert\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735407 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735393 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-envoy\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735419 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:50:09.735683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735438 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:50:09.735683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735457 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-podinfo\") pod \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\" (UID: \"915b4d84-1d93-44d7-8d3e-3c7e5708278d\") " Jul 12 16:50:09.735683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735621 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:50:09.735683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735644 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-data" (OuterVolumeSpecName: "istio-data") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:50:09.735964 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735910 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:09.735964 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735932 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-workload-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:09.735964 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735946 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-data\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:09.735964 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735960 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-credential-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:09.736175 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.735928 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:50:09.737563 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.737536 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-kube-api-access-bkhng" (OuterVolumeSpecName: "kube-api-access-bkhng") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "kube-api-access-bkhng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:50:09.737682 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.737637 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:50:09.737682 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.737643 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-token" (OuterVolumeSpecName: "istio-token") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:50:09.737958 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.737936 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "915b4d84-1d93-44d7-8d3e-3c7e5708278d" (UID: "915b4d84-1d93-44d7-8d3e-3c7e5708278d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 16:50:09.836613 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.836582 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:09.836613 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.836610 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkhng\" (UniqueName: \"kubernetes.io/projected/915b4d84-1d93-44d7-8d3e-3c7e5708278d-kube-api-access-bkhng\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:09.836613 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.836619 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istiod-ca-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:09.836885 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.836628 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-envoy\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:09.836885 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:09.836636 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915b4d84-1d93-44d7-8d3e-3c7e5708278d-istio-podinfo\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:50:10.195076 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.195021 2575 generic.go:356] "Generic (PLEG): container finished" podID="915b4d84-1d93-44d7-8d3e-3c7e5708278d" containerID="b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136" exitCode=0 Jul 12 16:50:10.195302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.195212 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" event={"ID":"915b4d84-1d93-44d7-8d3e-3c7e5708278d","Type":"ContainerDied","Data":"b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136"} Jul 12 16:50:10.195382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.195303 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" event={"ID":"915b4d84-1d93-44d7-8d3e-3c7e5708278d","Type":"ContainerDied","Data":"8f5c570c22041554fea2610f12f071f72a0e6a79c439d1652ce1fe0284bafe0a"} Jul 12 16:50:10.195382 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.195344 2575 scope.go:117] "RemoveContainer" containerID="b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136" Jul 12 16:50:10.195606 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.195564 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k" Jul 12 16:50:10.208685 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.208663 2575 scope.go:117] "RemoveContainer" containerID="b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136" Jul 12 16:50:10.208940 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:50:10.208921 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136\": container with ID starting with b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136 not found: ID does not exist" containerID="b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136" Jul 12 16:50:10.208993 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.208949 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136"} err="failed to get container status \"b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136\": rpc error: code = NotFound desc = could not find container \"b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136\": container with ID starting with b3504c64f79cabd134da45e7aebee76882ddc8e528610043b5f5c2aa0f487136 not found: ID does not exist" Jul 12 16:50:10.215685 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.215662 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k"] Jul 12 16:50:10.220938 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.220918 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b44a9a33-openshift-default-7fc77db547-k8m7k"] Jul 12 16:50:10.243038 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.243000 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:10.303426 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.303389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:10.726559 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.726518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:10.826909 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.826872 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:10.931334 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:10.931289 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:11.875219 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:11.875169 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915b4d84-1d93-44d7-8d3e-3c7e5708278d" path="/var/lib/kubelet/pods/915b4d84-1d93-44d7-8d3e-3c7e5708278d/volumes" Jul 12 16:50:12.278930 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:12.278900 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:17.970999 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:17.970963 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:18.077934 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:18.077897 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:18.189203 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:18.189169 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:18.321465 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:18.321359 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:18.732893 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:18.732860 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:44.708258 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:44.708221 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:50.477170 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:50.477130 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:50.540914 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:50.540864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:50.622717 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:50.622654 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:50:50.721595 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:50:50.721561 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:06.600133 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:06.600097 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:12.399182 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:12.399139 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:12.470510 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:12.470468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:12.536062 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:12.536029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:12.635591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:12.635550 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:23.171484 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:23.171446 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:24.395152 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:24.395112 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:29.724818 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:29.724779 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:29.828096 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:29.828058 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:29.925165 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:29.925123 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:30.025447 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:30.025349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:30.762442 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:30.762409 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:30.862559 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:30.862527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:30.963002 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:30.962965 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:32.627787 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:32.627734 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:33.191423 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:33.191383 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:33.266629 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:33.266590 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:51:33.267942 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:33.267901 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:38.272383 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:38.272344 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:38.372517 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:38.372480 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:38.958155 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:38.958107 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:42.460398 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:42.460368 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:51:44.936609 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:44.936569 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:45.037233 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.037183 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:45.175710 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.175672 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:45.442597 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.442569 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782"] Jul 12 16:51:45.442966 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.442954 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915b4d84-1d93-44d7-8d3e-3c7e5708278d" containerName="istio-proxy" Jul 12 16:51:45.443024 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.442967 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="915b4d84-1d93-44d7-8d3e-3c7e5708278d" containerName="istio-proxy" Jul 12 16:51:45.443065 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.443057 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="915b4d84-1d93-44d7-8d3e-3c7e5708278d" containerName="istio-proxy" Jul 12 16:51:45.446361 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.446344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.449248 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.449230 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-048a61f7-openshift-default-dockercfg-xcwgh\"" Jul 12 16:51:45.469003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.468981 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782"] Jul 12 16:51:45.470667 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.470637 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:45.532371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-credential-socket\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.532533 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532375 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-certs\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.532533 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-token\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.532613 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-envoy\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.532613 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlzz\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-kube-api-access-ljlzz\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.532686 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-socket\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.532730 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istiod-ca-cert\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.532776 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-data\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.532817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.532802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-podinfo\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.633885 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.633853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istiod-ca-cert\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634057 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.633896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-data\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634057 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.633925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-podinfo\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-credential-socket\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-certs\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-token\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634478 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-data\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634543 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634514 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-credential-socket\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634543 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-envoy\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634656 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlzz\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-kube-api-access-ljlzz\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634656 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-socket\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634812 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-certs\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.634933 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.634920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-socket\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.635356 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.635338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istiod-ca-cert\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.636638 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.636621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-envoy\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.636698 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.636684 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-podinfo\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.644568 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.644531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-token\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.644784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.644759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlzz\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-kube-api-access-ljlzz\") pod \"e2e-mt-048a61f7-openshift-default-7c988d594c-7d782\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.676472 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.676443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:45.756428 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.756338 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:45.904387 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.904355 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782"] Jul 12 16:51:45.906274 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:51:45.906242 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0565d14_a436_4e4d_ba9f_7fb437ae0edd.slice/crio-e9c9ccfafadea825b14d1aa607e28981830702579ef200278969bfac4d649432 WatchSource:0}: Error finding container e9c9ccfafadea825b14d1aa607e28981830702579ef200278969bfac4d649432: Status 404 returned error can't find the container with id e9c9ccfafadea825b14d1aa607e28981830702579ef200278969bfac4d649432 Jul 12 16:51:45.908589 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.908554 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:51:45.908683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.908623 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:51:45.908683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:45.908650 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:51:46.723807 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:46.723774 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" event={"ID":"b0565d14-a436-4e4d-ba9f-7fb437ae0edd","Type":"ContainerStarted","Data":"334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9"} Jul 12 16:51:46.724231 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:46.723822 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" event={"ID":"b0565d14-a436-4e4d-ba9f-7fb437ae0edd","Type":"ContainerStarted","Data":"e9c9ccfafadea825b14d1aa607e28981830702579ef200278969bfac4d649432"} Jul 12 16:51:46.744028 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:46.743971 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" podStartSLOduration=1.74395448 podStartE2EDuration="1.74395448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:51:46.742948838 +0000 UTC m=+1605.512376732" watchObservedRunningTime="2026-07-12 16:51:46.74395448 +0000 UTC m=+1605.513382374" Jul 12 16:51:46.756906 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:46.756874 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:46.761636 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:46.761611 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:47.729292 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:47.729257 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:47.730284 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:47.730256 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:51:48.676740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:48.676679 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:48.776155 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:48.776112 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:48.876519 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:48.876484 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:49.329134 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:49.329096 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:49.428585 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:49.428544 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:49.530442 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:49.530407 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:51.362452 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:51.362416 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:51:53.200154 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:53.200112 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:53.870672 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:53.870610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:53.968632 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:53.968581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:54.336732 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:54.336695 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:51:59.798787 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:59.798752 2575 generic.go:356] "Generic (PLEG): container finished" podID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerID="334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9" exitCode=0 Jul 12 16:51:59.799236 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:59.798828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" event={"ID":"b0565d14-a436-4e4d-ba9f-7fb437ae0edd","Type":"ContainerDied","Data":"334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9"} Jul 12 16:51:59.799306 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:59.799290 2575 scope.go:117] "RemoveContainer" containerID="334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9" Jul 12 16:51:59.800297 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:59.800268 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:51:59.800373 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:59.800355 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:51:59.800417 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:51:59.800405 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:52:00.172491 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:00.172390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:00.235265 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:00.235225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:00.606225 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:00.606161 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:00.806475 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:00.806432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" event={"ID":"b0565d14-a436-4e4d-ba9f-7fb437ae0edd","Type":"ContainerStarted","Data":"9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f"} Jul 12 16:52:00.966249 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:00.966212 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:01.069124 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:01.069092 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:01.756663 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:01.756630 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:52:01.757895 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:01.757868 2575 patch_prober.go:28] interesting pod/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 12 16:52:01.758006 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:01.757921 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 12 16:52:02.729972 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:02.729934 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:52:02.757239 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:02.757180 2575 patch_prober.go:28] interesting pod/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 12 16:52:02.757405 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:02.757257 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 12 16:52:03.757493 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:03.757458 2575 patch_prober.go:28] interesting pod/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 12 16:52:03.757886 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:03.757523 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 12 16:52:04.756991 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:04.756943 2575 patch_prober.go:28] interesting pod/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 12 16:52:04.757168 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:04.757010 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 12 16:52:05.068579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:05.068500 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:52:05.760672 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:05.760635 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:52:05.761837 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:05.761817 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:52:06.808987 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:06.808949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:06.870315 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:06.870262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:07.057170 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:07.057138 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782"] Jul 12 16:52:07.057392 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:07.057342 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" containerID="cri-o://9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f" gracePeriod=30 Jul 12 16:52:07.077916 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:07.077837 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:07.372865 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:07.372770 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:07.488862 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:07.488825 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:07.587090 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:07.587057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:07.687417 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:07.687384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:12.322426 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.322357 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:52:12.415849 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.415814 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-socket\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.415864 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljlzz\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-kube-api-access-ljlzz\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.415897 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-credential-socket\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.415935 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-podinfo\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416008 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-token\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416306 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416072 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istiod-ca-cert\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416306 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416115 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-data\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416306 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416145 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-certs\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416306 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416175 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-envoy\") pod \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\" (UID: \"b0565d14-a436-4e4d-ba9f-7fb437ae0edd\") " Jul 12 16:52:12.416306 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416223 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:52:12.416306 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416273 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:52:12.416601 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416521 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.416601 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416541 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-credential-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.416708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416616 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:52:12.416817 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.416798 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:52:12.417293 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.417265 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-data" (OuterVolumeSpecName: "istio-data") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:52:12.418356 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.418329 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-kube-api-access-ljlzz" (OuterVolumeSpecName: "kube-api-access-ljlzz") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "kube-api-access-ljlzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:52:12.418356 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.418348 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 16:52:12.418554 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.418535 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:52:12.418606 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.418571 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-token" (OuterVolumeSpecName: "istio-token") pod "b0565d14-a436-4e4d-ba9f-7fb437ae0edd" (UID: "b0565d14-a436-4e4d-ba9f-7fb437ae0edd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:52:12.517276 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.517235 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istiod-ca-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.517276 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.517272 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-data\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.517276 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.517285 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-workload-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.517511 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.517296 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-envoy\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.517511 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.517304 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlzz\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-kube-api-access-ljlzz\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.517511 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.517317 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-podinfo\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.517511 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.517328 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0565d14-a436-4e4d-ba9f-7fb437ae0edd-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:52:12.892702 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.892669 2575 generic.go:356] "Generic (PLEG): container finished" podID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerID="9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f" exitCode=0 Jul 12 16:52:12.892880 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.892718 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" Jul 12 16:52:12.892880 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.892749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" event={"ID":"b0565d14-a436-4e4d-ba9f-7fb437ae0edd","Type":"ContainerDied","Data":"9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f"} Jul 12 16:52:12.892880 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.892807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782" event={"ID":"b0565d14-a436-4e4d-ba9f-7fb437ae0edd","Type":"ContainerDied","Data":"e9c9ccfafadea825b14d1aa607e28981830702579ef200278969bfac4d649432"} Jul 12 16:52:12.892880 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.892838 2575 scope.go:117] "RemoveContainer" containerID="9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f" Jul 12 16:52:12.902393 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.902376 2575 scope.go:117] "RemoveContainer" containerID="334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9" Jul 12 16:52:12.910640 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.910619 2575 scope.go:117] "RemoveContainer" containerID="9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f" Jul 12 16:52:12.910893 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:52:12.910868 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f\": container with ID starting with 9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f not found: ID does not exist" containerID="9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f" Jul 12 16:52:12.910962 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.910906 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f"} err="failed to get container status \"9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f\": rpc error: code = NotFound desc = could not find container \"9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f\": container with ID starting with 9c70a4a0346dcf74ef7b6460c00c27f46d9b6a6094c59751a6b7bc15640fb29f not found: ID does not exist" Jul 12 16:52:12.910962 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.910931 2575 scope.go:117] "RemoveContainer" containerID="334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9" Jul 12 16:52:12.911218 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:52:12.911177 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9\": container with ID starting with 334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9 not found: ID does not exist" containerID="334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9" Jul 12 16:52:12.911266 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.911226 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9"} err="failed to get container status \"334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9\": rpc error: code = NotFound desc = could not find container \"334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9\": container with ID starting with 334c6b856cd4e507838a7bbbd99a2f9a58acef4432e89d1c194f9d02676c59b9 not found: ID does not exist" Jul 12 16:52:12.914993 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.914976 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782"] Jul 12 16:52:12.917408 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:12.917387 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-048a61f7-openshift-default-7c988d594c-7d782"] Jul 12 16:52:13.245446 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:13.245403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:13.348676 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:13.348639 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:13.463423 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:13.463391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:13.646117 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:13.646023 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:13.876312 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:13.876281 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" path="/var/lib/kubelet/pods/b0565d14-a436-4e4d-ba9f-7fb437ae0edd/volumes" Jul 12 16:52:19.141766 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:19.141724 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:19.242309 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:19.242275 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:26.041085 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.041044 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:26.327385 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.327292 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr"] Jul 12 16:52:26.327800 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.327781 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" Jul 12 16:52:26.327894 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.327803 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" Jul 12 16:52:26.327894 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.327831 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" Jul 12 16:52:26.327894 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.327839 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" Jul 12 16:52:26.328056 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.327952 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" Jul 12 16:52:26.328056 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.327966 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0565d14-a436-4e4d-ba9f-7fb437ae0edd" containerName="istio-proxy" Jul 12 16:52:26.332504 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.332481 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.334797 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.334778 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-44bf2bf1-openshift-default-dockercfg-6xhpg\"" Jul 12 16:52:26.343231 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.343206 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr"] Jul 12 16:52:26.355488 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.355464 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:26.443459 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-credential-socket\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.443627 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-socket\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.443627 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117ca94d-a185-4019-8844-f200336ec813-istiod-ca-cert\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.443627 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2bh\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-kube-api-access-wk2bh\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.443627 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-envoy\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.443772 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-istio-token\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.443772 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117ca94d-a185-4019-8844-f200336ec813-istio-podinfo\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.443837 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-certs\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.443837 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.443813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-data\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.544542 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-envoy\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.544542 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-istio-token\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.544750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117ca94d-a185-4019-8844-f200336ec813-istio-podinfo\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.544750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-certs\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.544750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-data\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.544750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-credential-socket\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.544750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-socket\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.545023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117ca94d-a185-4019-8844-f200336ec813-istiod-ca-cert\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.545023 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.544810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2bh\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-kube-api-access-wk2bh\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.545143 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.545083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-certs\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.545143 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.545118 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-data\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.545296 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.545183 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-credential-socket\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.545442 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.545422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-socket\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.545502 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.545489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117ca94d-a185-4019-8844-f200336ec813-istiod-ca-cert\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.546834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.546805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-envoy\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.547059 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.547043 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117ca94d-a185-4019-8844-f200336ec813-istio-podinfo\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.552256 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.552234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-istio-token\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.552333 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.552238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2bh\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-kube-api-access-wk2bh\") pod \"e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.643308 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.643220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:26.702041 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.701923 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:26.791213 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.791169 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr"] Jul 12 16:52:26.792889 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:52:26.792867 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117ca94d_a185_4019_8844_f200336ec813.slice/crio-38f2dd760e9636a760cebcbd0c0643811d3987fb93bb20e224d5a6f8c881d38b WatchSource:0}: Error finding container 38f2dd760e9636a760cebcbd0c0643811d3987fb93bb20e224d5a6f8c881d38b: Status 404 returned error can't find the container with id 38f2dd760e9636a760cebcbd0c0643811d3987fb93bb20e224d5a6f8c881d38b Jul 12 16:52:26.795184 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.795153 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:52:26.795285 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.795244 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:52:26.795353 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.795290 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:52:26.978781 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.978748 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" event={"ID":"117ca94d-a185-4019-8844-f200336ec813","Type":"ContainerStarted","Data":"1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6"} Jul 12 16:52:26.978987 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:26.978821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" event={"ID":"117ca94d-a185-4019-8844-f200336ec813","Type":"ContainerStarted","Data":"38f2dd760e9636a760cebcbd0c0643811d3987fb93bb20e224d5a6f8c881d38b"} Jul 12 16:52:27.000585 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:27.000523 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" podStartSLOduration=1.000503965 podStartE2EDuration="1.000503965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:52:26.996002657 +0000 UTC m=+1645.765430554" watchObservedRunningTime="2026-07-12 16:52:27.000503965 +0000 UTC m=+1645.769931876" Jul 12 16:52:27.644386 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:27.644346 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:27.649708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:27.649682 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:27.985154 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:27.985126 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:27.985986 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:27.985969 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:28.907621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:28.907581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:28.993949 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:28.993906 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:29.051286 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:29.051247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:29.187455 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:29.187418 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:29.446924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:29.446832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:29.554025 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:29.553992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:33.414597 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:33.414560 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:34.370599 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:34.370562 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:52:42.074240 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:42.074203 2575 generic.go:356] "Generic (PLEG): container finished" podID="117ca94d-a185-4019-8844-f200336ec813" containerID="1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6" exitCode=0 Jul 12 16:52:42.074642 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:42.074215 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" event={"ID":"117ca94d-a185-4019-8844-f200336ec813","Type":"ContainerDied","Data":"1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6"} Jul 12 16:52:42.074693 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:42.074681 2575 scope.go:117] "RemoveContainer" containerID="1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6" Jul 12 16:52:42.075686 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:42.075659 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:52:42.075742 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:42.075728 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:52:42.075779 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:42.075760 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:52:42.985668 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:42.985631 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:43.081452 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:43.081412 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" event={"ID":"117ca94d-a185-4019-8844-f200336ec813","Type":"ContainerStarted","Data":"ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4"} Jul 12 16:52:43.643820 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:43.643785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:43.644876 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:43.644850 2575 patch_prober.go:28] interesting pod/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 12 16:52:43.645009 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:43.644904 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 12 16:52:44.643916 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:44.643880 2575 patch_prober.go:28] interesting pod/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 12 16:52:44.644325 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:44.643939 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 12 16:52:45.438409 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:45.438370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:45.644177 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:45.644140 2575 patch_prober.go:28] interesting pod/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 12 16:52:45.644686 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:45.644223 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 12 16:52:46.644379 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:46.644338 2575 patch_prober.go:28] interesting pod/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 12 16:52:46.644838 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:46.644400 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 12 16:52:47.648256 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:47.648224 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:47.648692 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:47.648670 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:47.649416 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:47.649400 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:52:49.804024 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:49.803992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:51.723631 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:51.723584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:51.824793 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:51.824758 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:52:53.261596 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:52:53.261562 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:53:19.064035 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:19.063998 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:53:21.166622 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:21.166581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:53:21.635793 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:21.635757 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:53:21.744645 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:21.744607 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:53:21.889545 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:21.889448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:53:27.278113 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:27.278026 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:53:32.350167 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:32.350124 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:53:33.460846 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:33.460811 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:53:50.035469 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:50.035429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:53:50.092266 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:53:50.092231 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:08.579092 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:08.579062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:09.317752 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:09.317710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:11.964137 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:11.964094 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:54:14.902456 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:14.902421 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:15.005856 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:15.005813 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:15.213500 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:15.213471 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr"] Jul 12 16:54:15.262575 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:15.262541 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:15.452563 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:15.452527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:15.601421 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:15.601308 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" containerID="cri-o://ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4" gracePeriod=30 Jul 12 16:54:16.508861 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:16.508822 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:16.611363 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:16.611327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:16.709524 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:16.709481 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:17.446810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:17.446779 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:17.549038 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:17.549004 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:17.646794 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:17.646756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:17.650717 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:17.650687 2575 patch_prober.go:28] interesting pod/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 16:54:17.650877 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:17.650749 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 16:54:19.041078 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:19.041042 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:20.857430 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.857404 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:54:20.936062 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936030 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-data\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936288 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936102 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-envoy\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936288 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936177 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-certs\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936288 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936261 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk2bh\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-kube-api-access-wk2bh\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936499 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936298 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-socket\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936499 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936327 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117ca94d-a185-4019-8844-f200336ec813-istiod-ca-cert\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936499 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936364 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-credential-socket\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936499 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936392 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117ca94d-a185-4019-8844-f200336ec813-istio-podinfo\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936499 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936421 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-istio-token\") pod \"117ca94d-a185-4019-8844-f200336ec813\" (UID: \"117ca94d-a185-4019-8844-f200336ec813\") " Jul 12 16:54:20.936499 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936461 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:54:20.936800 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936584 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:54:20.936800 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936784 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:54:20.936929 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936790 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:20.936929 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936827 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-workload-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:20.937033 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.936944 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117ca94d-a185-4019-8844-f200336ec813-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:54:20.937124 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.937093 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-data" (OuterVolumeSpecName: "istio-data") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:54:20.938447 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.938416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-kube-api-access-wk2bh" (OuterVolumeSpecName: "kube-api-access-wk2bh") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "kube-api-access-wk2bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:54:20.938568 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.938547 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-istio-token" (OuterVolumeSpecName: "istio-token") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:54:20.938762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.938736 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/117ca94d-a185-4019-8844-f200336ec813-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 16:54:20.938762 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:20.938747 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "117ca94d-a185-4019-8844-f200336ec813" (UID: "117ca94d-a185-4019-8844-f200336ec813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:54:21.038126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.038089 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-credential-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:21.038126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.038119 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117ca94d-a185-4019-8844-f200336ec813-istio-podinfo\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:21.038126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.038128 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:21.038126 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.038136 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-data\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:21.038397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.038144 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117ca94d-a185-4019-8844-f200336ec813-istio-envoy\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:21.038397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.038152 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk2bh\" (UniqueName: \"kubernetes.io/projected/117ca94d-a185-4019-8844-f200336ec813-kube-api-access-wk2bh\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:21.038397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.038160 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117ca94d-a185-4019-8844-f200336ec813-istiod-ca-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:54:21.636201 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.636163 2575 generic.go:356] "Generic (PLEG): container finished" podID="117ca94d-a185-4019-8844-f200336ec813" containerID="ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4" exitCode=0 Jul 12 16:54:21.636371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.636241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" Jul 12 16:54:21.636371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.636257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" event={"ID":"117ca94d-a185-4019-8844-f200336ec813","Type":"ContainerDied","Data":"ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4"} Jul 12 16:54:21.636371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.636313 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr" event={"ID":"117ca94d-a185-4019-8844-f200336ec813","Type":"ContainerDied","Data":"38f2dd760e9636a760cebcbd0c0643811d3987fb93bb20e224d5a6f8c881d38b"} Jul 12 16:54:21.636371 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.636335 2575 scope.go:117] "RemoveContainer" containerID="ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4" Jul 12 16:54:21.646572 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.646555 2575 scope.go:117] "RemoveContainer" containerID="1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6" Jul 12 16:54:21.655256 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.655239 2575 scope.go:117] "RemoveContainer" containerID="ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4" Jul 12 16:54:21.655518 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:54:21.655499 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4\": container with ID starting with ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4 not found: ID does not exist" containerID="ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4" Jul 12 16:54:21.655568 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.655527 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4"} err="failed to get container status \"ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4\": rpc error: code = NotFound desc = could not find container \"ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4\": container with ID starting with ced8423f5b891f2b0d586556a8115f8a0032af69102e955f35d4e6f2f465e0d4 not found: ID does not exist" Jul 12 16:54:21.655568 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.655546 2575 scope.go:117] "RemoveContainer" containerID="1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6" Jul 12 16:54:21.655811 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:54:21.655794 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6\": container with ID starting with 1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6 not found: ID does not exist" containerID="1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6" Jul 12 16:54:21.655872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.655814 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6"} err="failed to get container status \"1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6\": rpc error: code = NotFound desc = could not find container \"1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6\": container with ID starting with 1f5c92f91765b36c28b3c021da0a8c79f9159c7730fc248231ebcb27341b1bc6 not found: ID does not exist" Jul 12 16:54:21.658828 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.658807 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr"] Jul 12 16:54:21.664562 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.664544 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-44bf2bf1-openshift-default-67db459855-rh6zr"] Jul 12 16:54:21.875450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:21.875408 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="117ca94d-a185-4019-8844-f200336ec813" path="/var/lib/kubelet/pods/117ca94d-a185-4019-8844-f200336ec813/volumes" Jul 12 16:54:24.818855 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:24.818821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:24.879124 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:24.879094 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:25.069222 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:25.069115 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:54:25.506096 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:25.506062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:31.501515 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:31.501479 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:31.605465 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:31.605427 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:31.701183 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:31.701146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:34.463604 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:34.463571 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:54:42.196260 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:42.196221 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:54:42.546946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:54:42.546835 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:01.925179 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:01.925144 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:55:01.928767 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:01.928746 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 16:55:03.522121 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:03.522082 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:09.356307 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.356272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:09.456218 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.456157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:09.709704 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.709671 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:09.993470 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.993393 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs"] Jul 12 16:55:09.993825 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.993811 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" Jul 12 16:55:09.993872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.993828 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" Jul 12 16:55:09.993906 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.993900 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" Jul 12 16:55:09.996959 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.996941 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:09.999445 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:09.999429 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2aa2dc-openshift-default-dockercfg-tdztx\"" Jul 12 16:55:10.007897 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.007861 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs"] Jul 12 16:55:10.014300 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.014272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:10.087433 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-token\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.087614 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-certs\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.087614 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2nrg\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-kube-api-access-r2nrg\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.087614 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-envoy\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.087614 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-data\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.087773 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-podinfo\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.087773 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-credential-socket\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.087773 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-socket\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.087773 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.087729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istiod-ca-cert\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.188501 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.188471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-podinfo\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.188676 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.188527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-credential-socket\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.188676 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.188574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-socket\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.188676 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.188596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istiod-ca-cert\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.188676 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.188647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-token\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.188886 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.188681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-certs\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.188886 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.188712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2nrg\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-kube-api-access-r2nrg\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.189037 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.189001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-envoy\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.189184 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.189049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-data\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.189184 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.189010 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-socket\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.189184 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.189130 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-credential-socket\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.189184 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.189208 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-certs\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.189450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.189362 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-data\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.189450 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.189418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istiod-ca-cert\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.191021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.191000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-podinfo\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.191105 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.191031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-envoy\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.196554 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.196526 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:10.196654 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.196631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-token\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.197106 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.197089 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2nrg\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-kube-api-access-r2nrg\") pod \"e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.308810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.308727 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:10.450499 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.450474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs"] Jul 12 16:55:10.451499 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:55:10.451472 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8fc056_c318_4b24_b9f6_b6b52a58614b.slice/crio-56030439cd059e1a656e079a7fe05816c73ef9d207bde310b7345b983a7f51ec WatchSource:0}: Error finding container 56030439cd059e1a656e079a7fe05816c73ef9d207bde310b7345b983a7f51ec: Status 404 returned error can't find the container with id 56030439cd059e1a656e079a7fe05816c73ef9d207bde310b7345b983a7f51ec Jul 12 16:55:10.454064 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.454029 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:10.454172 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.454111 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:10.454172 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.454154 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:10.915377 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.915334 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" event={"ID":"5e8fc056-c318-4b24-b9f6-b6b52a58614b","Type":"ContainerStarted","Data":"d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173"} Jul 12 16:55:10.915571 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.915386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" event={"ID":"5e8fc056-c318-4b24-b9f6-b6b52a58614b","Type":"ContainerStarted","Data":"56030439cd059e1a656e079a7fe05816c73ef9d207bde310b7345b983a7f51ec"} Jul 12 16:55:10.941997 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:10.941943 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" podStartSLOduration=1.941923424 podStartE2EDuration="1.941923424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:55:10.940662793 +0000 UTC m=+1809.710090690" watchObservedRunningTime="2026-07-12 16:55:10.941923424 +0000 UTC m=+1809.711351320" Jul 12 16:55:11.309538 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:11.309503 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:11.315957 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:11.315929 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:11.924503 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:11.924464 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:11.926517 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:11.926488 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:12.923332 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:12.923294 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:13.014172 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:13.014132 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:13.110835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:13.110797 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:13.230916 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:13.230881 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:13.490774 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:13.490686 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:13.655602 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:13.655565 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:17.425607 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:17.425568 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:17.461570 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:17.461537 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:55:17.746227 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:17.746165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:18.026832 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.026745 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt"] Jul 12 16:55:18.027286 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.027263 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" Jul 12 16:55:18.027286 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.027280 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" Jul 12 16:55:18.027448 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.027350 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="117ca94d-a185-4019-8844-f200336ec813" containerName="istio-proxy" Jul 12 16:55:18.030659 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.030637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.033420 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.033400 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c3601d-openshift-default-dockercfg-sdb7s\"" Jul 12 16:55:18.042337 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.042309 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt"] Jul 12 16:55:18.057469 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.057429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:18.167397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-istio-token\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.167579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-data\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.167579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-certs\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.167579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-credential-socket\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.167579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-socket\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.167750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5k2\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-kube-api-access-hr5k2\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.167750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-envoy\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.167750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8d959b-f706-4a72-b8af-350249302fb9-istiod-ca-cert\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.167750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.167707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8d959b-f706-4a72-b8af-350249302fb9-istio-podinfo\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.268809 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.268773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-certs\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.268821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-credential-socket\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.268848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-socket\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.268885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5k2\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-kube-api-access-hr5k2\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.268928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-envoy\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269003 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.268951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8d959b-f706-4a72-b8af-350249302fb9-istiod-ca-cert\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269300 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.269007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8d959b-f706-4a72-b8af-350249302fb9-istio-podinfo\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269300 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.269054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-istio-token\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269300 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.269095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-data\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269300 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.269176 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-certs\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269300 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.269216 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-socket\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.269462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-credential-socket\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269561 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.269507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-data\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.269939 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.269918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8d959b-f706-4a72-b8af-350249302fb9-istiod-ca-cert\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.271258 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.271237 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-envoy\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.271917 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.271894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8d959b-f706-4a72-b8af-350249302fb9-istio-podinfo\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.277217 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.277140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-istio-token\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.277338 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.277319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5k2\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-kube-api-access-hr5k2\") pod \"e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.337507 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.337471 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:18.342460 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.342434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:18.538642 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.538553 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt"] Jul 12 16:55:18.543459 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:55:18.543429 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8d959b_f706_4a72_b8af_350249302fb9.slice/crio-167f6cafe917bd0a52e9e3ca33ef35810dd5bb73c1841fc7a4d04ed2040e2ca8 WatchSource:0}: Error finding container 167f6cafe917bd0a52e9e3ca33ef35810dd5bb73c1841fc7a4d04ed2040e2ca8: Status 404 returned error can't find the container with id 167f6cafe917bd0a52e9e3ca33ef35810dd5bb73c1841fc7a4d04ed2040e2ca8 Jul 12 16:55:18.545398 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.545363 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:18.545496 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.545428 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:18.545496 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.545455 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:18.967441 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.967400 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" event={"ID":"ff8d959b-f706-4a72-b8af-350249302fb9","Type":"ContainerStarted","Data":"b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9"} Jul 12 16:55:18.967642 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.967459 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" event={"ID":"ff8d959b-f706-4a72-b8af-350249302fb9","Type":"ContainerStarted","Data":"167f6cafe917bd0a52e9e3ca33ef35810dd5bb73c1841fc7a4d04ed2040e2ca8"} Jul 12 16:55:18.992965 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:18.992921 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" podStartSLOduration=0.992904775 podStartE2EDuration="992.904775ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:55:18.991457956 +0000 UTC m=+1817.760885873" watchObservedRunningTime="2026-07-12 16:55:18.992904775 +0000 UTC m=+1817.762332669" Jul 12 16:55:19.342895 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:19.342806 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:19.347828 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:19.347799 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:19.973594 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:19.973563 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:19.974476 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:19.974455 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:20.992397 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:20.992361 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:21.088860 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:21.088821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:21.189540 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:21.189506 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:21.323340 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:21.323243 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:21.702457 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:21.702422 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:21.811227 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:21.811182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:25.366669 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:25.366586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:26.005621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:26.005588 2575 generic.go:356] "Generic (PLEG): container finished" podID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerID="d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173" exitCode=0 Jul 12 16:55:26.005796 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:26.005662 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" event={"ID":"5e8fc056-c318-4b24-b9f6-b6b52a58614b","Type":"ContainerDied","Data":"d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173"} Jul 12 16:55:26.006099 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:26.006083 2575 scope.go:117] "RemoveContainer" containerID="d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173" Jul 12 16:55:26.007305 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:26.007273 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:26.007416 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:26.007343 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:26.007416 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:26.007373 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:26.924835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:26.924802 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:27.017924 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:27.017881 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" event={"ID":"5e8fc056-c318-4b24-b9f6-b6b52a58614b","Type":"ContainerStarted","Data":"7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926"} Jul 12 16:55:27.310213 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:27.309678 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:27.311049 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:27.310734 2575 patch_prober.go:28] interesting pod/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 12 16:55:27.311049 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:27.310790 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 12 16:55:28.310262 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:28.310108 2575 patch_prober.go:28] interesting pod/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 12 16:55:28.310262 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:28.310207 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 12 16:55:29.309799 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:29.309758 2575 patch_prober.go:28] interesting pod/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 12 16:55:29.310014 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:29.309829 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 12 16:55:30.310290 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:30.310244 2575 patch_prober.go:28] interesting pod/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 12 16:55:30.310804 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:30.310334 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 12 16:55:31.315578 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:31.315540 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:31.316027 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:31.316004 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:31.316906 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:31.316882 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:55:31.573069 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:31.572970 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:55:32.206061 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:32.206031 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:34.073646 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:34.073607 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff8d959b-f706-4a72-b8af-350249302fb9" containerID="b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9" exitCode=0 Jul 12 16:55:34.074057 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:34.073682 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" event={"ID":"ff8d959b-f706-4a72-b8af-350249302fb9","Type":"ContainerDied","Data":"b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9"} Jul 12 16:55:34.074158 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:34.074140 2575 scope.go:117] "RemoveContainer" containerID="b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9" Jul 12 16:55:34.075251 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:34.075216 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:34.075312 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:34.075294 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:34.075349 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:34.075337 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:55:34.974101 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:34.974055 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:35.081202 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:35.081153 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" event={"ID":"ff8d959b-f706-4a72-b8af-350249302fb9","Type":"ContainerStarted","Data":"e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455"} Jul 12 16:55:35.221839 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:35.221805 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:35.324035 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:35.323939 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:35.342709 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:35.342671 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:35.343835 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:35.343808 2575 patch_prober.go:28] interesting pod/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 12 16:55:35.343963 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:35.343878 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 12 16:55:36.343831 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:36.343790 2575 patch_prober.go:28] interesting pod/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 12 16:55:36.344333 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:36.343863 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 12 16:55:37.343735 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:37.343694 2575 patch_prober.go:28] interesting pod/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 12 16:55:37.343911 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:37.343756 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 12 16:55:38.343253 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:38.343216 2575 patch_prober.go:28] interesting pod/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 12 16:55:38.343431 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:38.343281 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 12 16:55:39.347052 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:39.347026 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:39.347449 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:39.347383 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:39.347967 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:39.347949 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:55:42.463683 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:42.463649 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:55:42.496097 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:42.496063 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:55:42.596900 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:55:42.596870 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:56:01.489488 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:56:01.489436 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:56:06.861740 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:56:06.861705 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:56:24.063143 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:56:24.063051 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:56:25.118916 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:56:25.118879 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:56:26.258563 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:56:26.258519 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:56:27.059621 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:56:27.059589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:56:30.788083 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:56:30.788051 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:56:41.964798 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:56:41.964770 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:57:07.737012 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:07.736968 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:07.989592 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:07.989512 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs"] Jul 12 16:57:07.989810 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:07.989783 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" containerID="cri-o://7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926" gracePeriod=30 Jul 12 16:57:08.017204 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:08.017160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:08.258663 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:08.258573 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:08.407914 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:08.407883 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:08.822649 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:08.822610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:08.925290 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:08.925254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:09.025708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:09.025664 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:09.154041 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:09.153956 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:12.471560 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:12.471526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:57:13.252028 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.252006 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:57:13.388923 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.388838 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2nrg\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-kube-api-access-r2nrg\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.388923 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.388878 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-credential-socket\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.388923 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.388923 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istiod-ca-cert\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.389232 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389040 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-envoy\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.389232 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-data\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.389232 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389176 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-socket\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.389398 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389240 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:13.389398 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389250 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-token\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.389398 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389277 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-podinfo\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.389398 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389347 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-certs\") pod \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\" (UID: \"5e8fc056-c318-4b24-b9f6-b6b52a58614b\") " Jul 12 16:57:13.389591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389392 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:57:13.389591 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389454 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:13.389712 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389664 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:13.389767 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389757 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.389826 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389773 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-credential-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.389826 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389788 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istiod-ca-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.389826 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.389802 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-workload-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.390110 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.390083 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:13.391276 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.391243 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:13.391388 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.391338 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 16:57:13.391485 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.391466 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-kube-api-access-r2nrg" (OuterVolumeSpecName: "kube-api-access-r2nrg") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "kube-api-access-r2nrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:57:13.391606 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.391590 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e8fc056-c318-4b24-b9f6-b6b52a58614b" (UID: "5e8fc056-c318-4b24-b9f6-b6b52a58614b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:57:13.491292 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.491257 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-envoy\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.491292 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.491285 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-data\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.491292 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.491294 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.491717 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.491302 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8fc056-c318-4b24-b9f6-b6b52a58614b-istio-podinfo\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.491717 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.491311 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2nrg\" (UniqueName: \"kubernetes.io/projected/5e8fc056-c318-4b24-b9f6-b6b52a58614b-kube-api-access-r2nrg\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:13.648415 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.648321 2575 generic.go:356] "Generic (PLEG): container finished" podID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerID="7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926" exitCode=0 Jul 12 16:57:13.648415 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.648385 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" Jul 12 16:57:13.648415 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.648403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" event={"ID":"5e8fc056-c318-4b24-b9f6-b6b52a58614b","Type":"ContainerDied","Data":"7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926"} Jul 12 16:57:13.648698 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.648488 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs" event={"ID":"5e8fc056-c318-4b24-b9f6-b6b52a58614b","Type":"ContainerDied","Data":"56030439cd059e1a656e079a7fe05816c73ef9d207bde310b7345b983a7f51ec"} Jul 12 16:57:13.648698 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.648519 2575 scope.go:117] "RemoveContainer" containerID="7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926" Jul 12 16:57:13.658778 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.658760 2575 scope.go:117] "RemoveContainer" containerID="d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173" Jul 12 16:57:13.667335 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.667319 2575 scope.go:117] "RemoveContainer" containerID="7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926" Jul 12 16:57:13.667591 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:57:13.667574 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926\": container with ID starting with 7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926 not found: ID does not exist" containerID="7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926" Jul 12 16:57:13.667643 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.667601 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926"} err="failed to get container status \"7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926\": rpc error: code = NotFound desc = could not find container \"7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926\": container with ID starting with 7dd7705e240c774dfa4f505c03c4846b3c088af8f49a8d9e7e4c0108a8125926 not found: ID does not exist" Jul 12 16:57:13.667643 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.667618 2575 scope.go:117] "RemoveContainer" containerID="d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173" Jul 12 16:57:13.667858 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:57:13.667839 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173\": container with ID starting with d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173 not found: ID does not exist" containerID="d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173" Jul 12 16:57:13.667898 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.667865 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173"} err="failed to get container status \"d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173\": rpc error: code = NotFound desc = could not find container \"d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173\": container with ID starting with d707099e7e486ce0d0f7003171ccb25010ecc1deee17646ac75a54f12afff173 not found: ID does not exist" Jul 12 16:57:13.671811 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.671790 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs"] Jul 12 16:57:13.674298 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.674279 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2aa2dc-openshift-default-64c75fd6c5-bc5hs"] Jul 12 16:57:13.877059 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:13.877025 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" path="/var/lib/kubelet/pods/5e8fc056-c318-4b24-b9f6-b6b52a58614b/volumes" Jul 12 16:57:14.556520 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:14.556489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:14.626650 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:14.626614 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:14.809409 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:14.809322 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt"] Jul 12 16:57:14.809872 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:14.809835 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" containerID="cri-o://e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455" gracePeriod=30 Jul 12 16:57:14.827749 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:14.827717 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:15.053141 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:15.053103 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:15.381687 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:15.381644 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:15.687299 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:15.687266 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:15.987486 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:15.987393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:20.074991 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.074917 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:57:20.252947 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.252913 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr5k2\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-kube-api-access-hr5k2\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.252947 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.252950 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-socket\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.253302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.252968 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-data\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.253302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.252986 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-certs\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.253302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253014 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-istio-token\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.253302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253062 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8d959b-f706-4a72-b8af-350249302fb9-istio-podinfo\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.253302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253241 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-credential-socket\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.253302 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253295 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:20.253605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253311 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:20.253605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253342 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-envoy\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.253605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253372 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8d959b-f706-4a72-b8af-350249302fb9-istiod-ca-cert\") pod \"ff8d959b-f706-4a72-b8af-350249302fb9\" (UID: \"ff8d959b-f706-4a72-b8af-350249302fb9\") " Jul 12 16:57:20.253605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253526 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:20.253831 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253742 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-credential-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.253831 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253761 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.253831 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253776 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-workload-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.253831 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253824 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8d959b-f706-4a72-b8af-350249302fb9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:57:20.254040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.253855 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:20.255446 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.255421 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:57:20.255727 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.255695 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:57:20.255727 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.255698 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-kube-api-access-hr5k2" (OuterVolumeSpecName: "kube-api-access-hr5k2") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "kube-api-access-hr5k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:57:20.255864 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.255764 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff8d959b-f706-4a72-b8af-350249302fb9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff8d959b-f706-4a72-b8af-350249302fb9" (UID: "ff8d959b-f706-4a72-b8af-350249302fb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 16:57:20.354915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.354834 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-envoy\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.354915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.354862 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8d959b-f706-4a72-b8af-350249302fb9-istiod-ca-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.354915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.354871 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5k2\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-kube-api-access-hr5k2\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.354915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.354880 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8d959b-f706-4a72-b8af-350249302fb9-istio-data\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.354915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.354889 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8d959b-f706-4a72-b8af-350249302fb9-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.354915 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.354897 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8d959b-f706-4a72-b8af-350249302fb9-istio-podinfo\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:57:20.692313 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.692279 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff8d959b-f706-4a72-b8af-350249302fb9" containerID="e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455" exitCode=0 Jul 12 16:57:20.692482 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.692331 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" Jul 12 16:57:20.692482 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.692365 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" event={"ID":"ff8d959b-f706-4a72-b8af-350249302fb9","Type":"ContainerDied","Data":"e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455"} Jul 12 16:57:20.692482 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.692422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt" event={"ID":"ff8d959b-f706-4a72-b8af-350249302fb9","Type":"ContainerDied","Data":"167f6cafe917bd0a52e9e3ca33ef35810dd5bb73c1841fc7a4d04ed2040e2ca8"} Jul 12 16:57:20.692482 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.692452 2575 scope.go:117] "RemoveContainer" containerID="e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455" Jul 12 16:57:20.703099 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.703081 2575 scope.go:117] "RemoveContainer" containerID="b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9" Jul 12 16:57:20.711612 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.711591 2575 scope.go:117] "RemoveContainer" containerID="e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455" Jul 12 16:57:20.711859 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:57:20.711840 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455\": container with ID starting with e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455 not found: ID does not exist" containerID="e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455" Jul 12 16:57:20.711913 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.711869 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455"} err="failed to get container status \"e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455\": rpc error: code = NotFound desc = could not find container \"e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455\": container with ID starting with e5ae2e81cc2f83bc6b79d32c0a977ad132af0a002ebe99526a5a011879094455 not found: ID does not exist" Jul 12 16:57:20.711913 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.711889 2575 scope.go:117] "RemoveContainer" containerID="b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9" Jul 12 16:57:20.712105 ip-10-0-134-123 kubenswrapper[2575]: E0712 16:57:20.712084 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9\": container with ID starting with b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9 not found: ID does not exist" containerID="b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9" Jul 12 16:57:20.712157 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.712115 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9"} err="failed to get container status \"b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9\": rpc error: code = NotFound desc = could not find container \"b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9\": container with ID starting with b840a03fae8060b5dbedf53cc44b4ff159379e8147b90d2109d45dfe7d105fa9 not found: ID does not exist" Jul 12 16:57:20.717739 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.717715 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt"] Jul 12 16:57:20.721622 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:20.721600 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c3601d-openshift-default-856869bdd5-jgxqt"] Jul 12 16:57:21.611671 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:21.611627 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:21.809144 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:21.809109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:21.875304 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:21.875220 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" path="/var/lib/kubelet/pods/ff8d959b-f706-4a72-b8af-350249302fb9/volumes" Jul 12 16:57:21.923785 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:21.923750 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:22.241452 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:22.241415 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:22.412012 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:22.411966 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:22.512570 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:22.512469 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:25.065568 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:25.065531 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:57:26.102394 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.102364 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:26.556670 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.556627 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:26.836077 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836002 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs"] Jul 12 16:57:26.836444 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836419 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" Jul 12 16:57:26.836444 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836439 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" Jul 12 16:57:26.836605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836458 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" Jul 12 16:57:26.836605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836463 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" Jul 12 16:57:26.836605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836540 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" Jul 12 16:57:26.836605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836550 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" Jul 12 16:57:26.836605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836556 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" Jul 12 16:57:26.836605 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.836562 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" Jul 12 16:57:26.841323 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.841299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:26.843649 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.843633 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b30e16-openshift-default-dockercfg-z4jk8\"" Jul 12 16:57:26.851040 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.851017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs"] Jul 12 16:57:26.876477 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.876444 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:26.977369 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:26.977328 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:27.015627 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.015591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-socket\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.015820 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.015644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-envoy\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.015820 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.015714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-token\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.015952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.015833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnvl6\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-kube-api-access-wnvl6\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.015952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.015885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-data\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.015952 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.015939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-certs\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.016106 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.015973 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istiod-ca-cert\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.016106 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.016008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-credential-socket\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.016106 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.016037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-podinfo\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117079 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.116979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnvl6\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-kube-api-access-wnvl6\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117079 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-data\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117079 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-certs\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istiod-ca-cert\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-credential-socket\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-podinfo\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-socket\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-envoy\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-token\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.117708 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-data\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.118094 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-certs\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.118094 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117766 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-credential-socket\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.118094 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-socket\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.118094 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.117933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istiod-ca-cert\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.119763 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.119733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-podinfo\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.119862 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.119774 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-envoy\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.124726 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.124701 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-token\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.125323 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.125308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnvl6\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-kube-api-access-wnvl6\") pod \"e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.153793 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.153763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:27.294401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.294373 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs"] Jul 12 16:57:27.295921 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:57:27.295891 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ae0753_e0a5_426f_9e19_08d0a8ff78bf.slice/crio-00526dc0d2e3e52aa8ff6ece1575afb687371338827b74c75ad8672c0417c259 WatchSource:0}: Error finding container 00526dc0d2e3e52aa8ff6ece1575afb687371338827b74c75ad8672c0417c259: Status 404 returned error can't find the container with id 00526dc0d2e3e52aa8ff6ece1575afb687371338827b74c75ad8672c0417c259 Jul 12 16:57:27.298229 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.298158 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:57:27.298318 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.298274 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:57:27.298361 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.298317 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:57:27.734843 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.734808 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" event={"ID":"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf","Type":"ContainerStarted","Data":"40b5e84c16c35ffad9241d6ffddac1faf62a16ae96ed4d339b7c9d371703fb99"} Jul 12 16:57:27.735025 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.734862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" event={"ID":"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf","Type":"ContainerStarted","Data":"00526dc0d2e3e52aa8ff6ece1575afb687371338827b74c75ad8672c0417c259"} Jul 12 16:57:27.756262 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:27.756211 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" podStartSLOduration=1.756171325 podStartE2EDuration="1.756171325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:57:27.754882135 +0000 UTC m=+1946.524310049" watchObservedRunningTime="2026-07-12 16:57:27.756171325 +0000 UTC m=+1946.525599219" Jul 12 16:57:28.154793 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:28.154700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:28.159461 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:28.159435 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:28.741028 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:28.740998 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:28.742032 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:28.742009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:29.638433 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:29.638395 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:29.723626 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:29.723586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:29.934372 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:29.934333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:30.048219 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:30.048157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:30.482939 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:30.482889 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:30.579756 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:30.579723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:34.141463 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:34.141423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:37.559816 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:37.559778 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:57:42.162928 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:42.162896 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:42.558779 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:42.558745 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:42.659076 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:42.659041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:42.759892 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:42.759850 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:42.905259 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:42.905147 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:44.783372 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:44.783330 2575 patch_prober.go:28] interesting pod/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 12 16:57:44.783826 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:44.783400 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 12 16:57:45.860411 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:45.860335 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerID="40b5e84c16c35ffad9241d6ffddac1faf62a16ae96ed4d339b7c9d371703fb99" exitCode=0 Jul 12 16:57:45.860411 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:45.860397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" event={"ID":"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf","Type":"ContainerDied","Data":"40b5e84c16c35ffad9241d6ffddac1faf62a16ae96ed4d339b7c9d371703fb99"} Jul 12 16:57:45.860888 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:45.860788 2575 scope.go:117] "RemoveContainer" containerID="40b5e84c16c35ffad9241d6ffddac1faf62a16ae96ed4d339b7c9d371703fb99" Jul 12 16:57:45.861842 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:45.861812 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:57:45.861920 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:45.861906 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:57:45.861956 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:45.861943 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 12 16:57:46.868510 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:46.868477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" event={"ID":"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf","Type":"ContainerStarted","Data":"a08b535235362b4d8e227c6f2251a614b33a45261ca7f3271a9804fbd4d2949b"} Jul 12 16:57:47.155044 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:47.154943 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:47.155319 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:47.155295 2575 patch_prober.go:28] interesting pod/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 12 16:57:47.155401 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:47.155343 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 12 16:57:48.154781 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:48.154739 2575 patch_prober.go:28] interesting pod/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 12 16:57:48.155214 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:48.154819 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 12 16:57:48.635668 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:48.635628 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:48.736539 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:48.736505 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:48.926304 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:48.926274 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:49.141834 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:49.141807 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:49.154611 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:49.154579 2575 patch_prober.go:28] interesting pod/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 12 16:57:49.154771 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:49.154627 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 12 16:57:49.339941 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:49.339849 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:49.786431 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:49.786390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:49.911922 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:49.911887 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:50.011376 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:50.011336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:50.155202 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:50.155106 2575 patch_prober.go:28] interesting pod/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 12 16:57:50.155202 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:50.155157 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 12 16:57:50.238080 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:50.238047 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:51.157953 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:51.157926 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:51.158342 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:51.158209 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:51.158880 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:51.158861 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:57:52.444892 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:52.444855 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:52.546246 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:52.546212 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:53.371840 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:53.371759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:57:55.530893 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:55.530854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:55.630428 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:55.630387 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:55.766784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:55.766744 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs"] Jul 12 16:57:55.767027 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:55.766999 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" containerID="cri-o://a08b535235362b4d8e227c6f2251a614b33a45261ca7f3271a9804fbd4d2949b" gracePeriod=30 Jul 12 16:57:55.781812 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:55.781786 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:56.003248 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:56.003215 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:56.341821 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:56.341789 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:56.641369 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:56.641263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:56.742567 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:56.742530 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:59.755029 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:59.754990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:59.851392 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:59.851351 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:57:59.921791 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:57:59.921738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:00.147889 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:00.147800 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:00.340199 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:00.340166 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:00.450921 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:00.450882 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:00.952177 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:00.952137 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerID="a08b535235362b4d8e227c6f2251a614b33a45261ca7f3271a9804fbd4d2949b" exitCode=0 Jul 12 16:58:00.952603 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:00.952223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" event={"ID":"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf","Type":"ContainerDied","Data":"a08b535235362b4d8e227c6f2251a614b33a45261ca7f3271a9804fbd4d2949b"} Jul 12 16:58:00.952603 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:00.952296 2575 scope.go:117] "RemoveContainer" containerID="40b5e84c16c35ffad9241d6ffddac1faf62a16ae96ed4d339b7c9d371703fb99" Jul 12 16:58:01.031434 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.031408 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:58:01.150652 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150571 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-data\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.150652 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150615 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-token\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.150857 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150672 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-credential-socket\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.150857 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150704 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-envoy\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.150857 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150742 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-certs\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.150857 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150796 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istiod-ca-cert\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.150857 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150822 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-socket\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.150857 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150843 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-podinfo\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.151179 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150878 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnvl6\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-kube-api-access-wnvl6\") pod \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\" (UID: \"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf\") " Jul 12 16:58:01.151179 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.150953 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:58:01.151179 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.151060 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:58:01.151179 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.151178 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 16:58:01.151430 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.151286 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-credential-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.151430 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.151307 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.151430 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.151322 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istiod-ca-cert\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.151430 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.151368 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:58:01.151600 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.151558 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:58:01.152892 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.152868 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:58:01.153021 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.153004 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 16:58:01.153361 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.153337 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-kube-api-access-wnvl6" (OuterVolumeSpecName: "kube-api-access-wnvl6") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "kube-api-access-wnvl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 16:58:01.153361 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.153340 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" (UID: "c1ae0753-e0a5-426f-9e19-08d0a8ff78bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 16:58:01.252729 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.252690 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-envoy\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.252729 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.252723 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-workload-socket\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.252729 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.252732 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-podinfo\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.252970 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.252741 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnvl6\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-kube-api-access-wnvl6\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.252970 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.252751 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-data\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.252970 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.252759 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf-istio-token\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 16:58:01.961538 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.961515 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" Jul 12 16:58:01.961951 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.961552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs" event={"ID":"c1ae0753-e0a5-426f-9e19-08d0a8ff78bf","Type":"ContainerDied","Data":"00526dc0d2e3e52aa8ff6ece1575afb687371338827b74c75ad8672c0417c259"} Jul 12 16:58:01.961951 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.961606 2575 scope.go:117] "RemoveContainer" containerID="a08b535235362b4d8e227c6f2251a614b33a45261ca7f3271a9804fbd4d2949b" Jul 12 16:58:01.983109 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.983084 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs"] Jul 12 16:58:01.988512 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:01.988488 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b30e16-openshift-default-7bc89dc6d7-rvngs"] Jul 12 16:58:03.878005 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:03.877970 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" path="/var/lib/kubelet/pods/c1ae0753-e0a5-426f-9e19-08d0a8ff78bf/volumes" Jul 12 16:58:03.964589 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:03.964556 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:58:04.253750 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:04.253695 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:04.579241 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:04.579117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:04.887179 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:04.887075 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:05.036403 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:05.036373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:06.313204 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:06.313160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:07.828474 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:07.828434 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:07.901671 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:07.901635 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:08.065928 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:08.065893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:08.175004 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:08.174890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:08.580174 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:08.580137 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:08.680655 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:08.680615 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:12.313946 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:12.313904 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:13.168465 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:13.168425 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:58:22.123408 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:22.123361 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:22.223778 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:22.223743 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:29.299604 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:29.299568 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:29.402004 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:29.401963 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:31.262136 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262100 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l"] Jul 12 16:58:31.262695 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262675 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" Jul 12 16:58:31.262784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262698 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8d959b-f706-4a72-b8af-350249302fb9" containerName="istio-proxy" Jul 12 16:58:31.262784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262719 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" Jul 12 16:58:31.262784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262727 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8fc056-c318-4b24-b9f6-b6b52a58614b" containerName="istio-proxy" Jul 12 16:58:31.262784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262754 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" Jul 12 16:58:31.262784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262763 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" Jul 12 16:58:31.262784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262775 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" Jul 12 16:58:31.262784 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262783 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" Jul 12 16:58:31.263140 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262870 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" Jul 12 16:58:31.263140 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.262886 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ae0753-e0a5-426f-9e19-08d0a8ff78bf" containerName="istio-proxy" Jul 12 16:58:31.266202 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.266167 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.269850 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.269830 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"kube-root-ca.crt\"" Jul 12 16:58:31.270017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.269835 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"default-dockercfg-ljksn\"" Jul 12 16:58:31.270017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.269836 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"openshift-service-ca.crt\"" Jul 12 16:58:31.270017 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.269934 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"auth-test-model-17d27e-kserve-self-signed-certs\"" Jul 12 16:58:31.273721 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.273700 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l"] Jul 12 16:58:31.284149 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.284117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:31.324476 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.324444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-dshm\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.324668 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.324491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tmp-dir\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.324668 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.324625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-home\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.324792 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.324701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjtg\" (UniqueName: \"kubernetes.io/projected/158e75df-d0d4-4538-ab68-bb56ff3dd2da-kube-api-access-dqjtg\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.324792 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.324733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tls-certs\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.324792 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.324764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-model-cache\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.425534 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.425496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tmp-dir\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.425734 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.425566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-home\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.425734 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.425616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjtg\" (UniqueName: \"kubernetes.io/projected/158e75df-d0d4-4538-ab68-bb56ff3dd2da-kube-api-access-dqjtg\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.425734 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.425635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tls-certs\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.425734 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.425654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-model-cache\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.425734 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.425704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-dshm\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.425996 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.425954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tmp-dir\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.426062 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.426027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-home\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.426062 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.426046 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-model-cache\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.428033 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.428014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-dshm\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.428229 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.428214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tls-certs\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.433856 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.433839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjtg\" (UniqueName: \"kubernetes.io/projected/158e75df-d0d4-4538-ab68-bb56ff3dd2da-kube-api-access-dqjtg\") pod \"auth-test-model-17d27e-kserve-85bc658c5-9nv8l\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.576532 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.576442 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:31.712706 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:31.712676 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l"] Jul 12 16:58:31.713819 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:58:31.713789 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod158e75df_d0d4_4538_ab68_bb56ff3dd2da.slice/crio-dbe6ff3fdb14bdafd537d05d0d761ced3103891b90af7986fb28145d2b1fd07a WatchSource:0}: Error finding container dbe6ff3fdb14bdafd537d05d0d761ced3103891b90af7986fb28145d2b1fd07a: Status 404 returned error can't find the container with id dbe6ff3fdb14bdafd537d05d0d761ced3103891b90af7986fb28145d2b1fd07a Jul 12 16:58:32.129400 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:32.129363 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" event={"ID":"158e75df-d0d4-4538-ab68-bb56ff3dd2da","Type":"ContainerStarted","Data":"fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4"} Jul 12 16:58:32.129579 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:32.129411 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" event={"ID":"158e75df-d0d4-4538-ab68-bb56ff3dd2da","Type":"ContainerStarted","Data":"dbe6ff3fdb14bdafd537d05d0d761ced3103891b90af7986fb28145d2b1fd07a"} Jul 12 16:58:32.150347 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:32.150300 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" podStartSLOduration=1.150284115 podStartE2EDuration="1.150284115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:58:32.147211161 +0000 UTC m=+2010.916639046" watchObservedRunningTime="2026-07-12 16:58:32.150284115 +0000 UTC m=+2010.919712043" Jul 12 16:58:34.673210 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:34.673157 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:58:41.577434 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:41.577395 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:41.577434 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:41.577445 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:41.590080 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:41.590049 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:42.196498 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:42.196464 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 16:58:56.778771 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.778720 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj"] Jul 12 16:58:56.782806 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.782785 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.786694 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.786670 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"auth-test-model-cc457c-kserve-self-signed-certs\"" Jul 12 16:58:56.786831 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.786702 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"openshift-service-ca.crt\"" Jul 12 16:58:56.786831 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.786725 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"default-dockercfg-m4v8t\"" Jul 12 16:58:56.786831 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.786820 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"kube-root-ca.crt\"" Jul 12 16:58:56.792944 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.792921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj"] Jul 12 16:58:56.800092 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.800067 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 16:58:56.869396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.869366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-dshm\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.869396 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.869400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-home\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.869616 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.869487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psmbq\" (UniqueName: \"kubernetes.io/projected/e5736868-a6ae-4478-a140-3d1801b44350-kube-api-access-psmbq\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.869616 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.869510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-model-cache\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.869616 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.869566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5736868-a6ae-4478-a140-3d1801b44350-tls-certs\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.869616 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.869595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-tmp-dir\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970206 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmbq\" (UniqueName: \"kubernetes.io/projected/e5736868-a6ae-4478-a140-3d1801b44350-kube-api-access-psmbq\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970388 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-model-cache\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970457 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5736868-a6ae-4478-a140-3d1801b44350-tls-certs\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970516 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-tmp-dir\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970604 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-dshm\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970646 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-home\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970821 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-model-cache\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970928 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-tmp-dir\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.970995 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.970968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-home\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.972725 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.972706 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-dshm\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.972914 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.972897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5736868-a6ae-4478-a140-3d1801b44350-tls-certs\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:56.980615 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:56.980586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmbq\" (UniqueName: \"kubernetes.io/projected/e5736868-a6ae-4478-a140-3d1801b44350-kube-api-access-psmbq\") pod \"auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:57.093221 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:57.093095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:58:57.234881 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:57.234846 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj"] Jul 12 16:58:57.236511 ip-10-0-134-123 kubenswrapper[2575]: W0712 16:58:57.236484 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5736868_a6ae_4478_a140_3d1801b44350.slice/crio-d2644a6bef09dd157bd373f7a740d9ae1ff16541bc7a3ec7cf9d15e29870040d WatchSource:0}: Error finding container d2644a6bef09dd157bd373f7a740d9ae1ff16541bc7a3ec7cf9d15e29870040d: Status 404 returned error can't find the container with id d2644a6bef09dd157bd373f7a740d9ae1ff16541bc7a3ec7cf9d15e29870040d Jul 12 16:58:57.278995 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:57.278962 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" event={"ID":"e5736868-a6ae-4478-a140-3d1801b44350","Type":"ContainerStarted","Data":"d2644a6bef09dd157bd373f7a740d9ae1ff16541bc7a3ec7cf9d15e29870040d"} Jul 12 16:58:58.287170 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:58.287132 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" event={"ID":"e5736868-a6ae-4478-a140-3d1801b44350","Type":"ContainerStarted","Data":"5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3"} Jul 12 16:58:58.304171 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:58:58.304119 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" podStartSLOduration=2.3041040649999998 podStartE2EDuration="2.304104065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 16:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 16:58:58.302945625 +0000 UTC m=+2037.072373522" watchObservedRunningTime="2026-07-12 16:58:58.304104065 +0000 UTC m=+2037.073531960" Jul 12 16:59:01.063147 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:59:01.063107 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:59:07.093909 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:59:07.093866 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:59:07.093909 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:59:07.093918 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:59:07.106495 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:59:07.106466 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:59:07.349721 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:59:07.349640 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 16:59:19.174274 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:59:19.174241 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:59:38.071862 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:59:38.071784 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 16:59:57.061469 ip-10-0-134-123 kubenswrapper[2575]: I0712 16:59:57.061431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:00:01.964119 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:01.964087 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 17:00:01.971312 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:01.971285 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zxb5k_934d23da-7dc2-40c1-84c1-9784f7a99dde/ovn-acl-logging/0.log" Jul 12 17:00:03.255120 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:03.255079 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:00:03.304528 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:03.304491 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:00:13.466772 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:13.466730 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:00:25.139421 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:25.139378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:00:30.284846 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:30.284801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:00:45.660214 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:45.660143 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:00:57.046099 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:57.046013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:00:57.099132 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:57.099105 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:00:59.351202 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:59.351155 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l"] Jul 12 17:00:59.369115 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:00:59.369079 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:01:00.264890 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:00.264861 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj"] Jul 12 17:01:00.283598 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:00.283569 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:01:03.858753 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:03.858719 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:01:07.062883 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.062843 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l"] Jul 12 17:01:07.066671 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.066644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.068879 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.068860 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"sub-test-model-17d27e-kserve-self-signed-certs\"" Jul 12 17:01:07.076406 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.076374 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l"] Jul 12 17:01:07.111040 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.110996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-home\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.111240 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.111073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-dshm\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.111240 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.111106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zt9l\" (UniqueName: \"kubernetes.io/projected/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-kube-api-access-9zt9l\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.111338 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.111246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-model-cache\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.111338 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.111280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tmp-dir\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.111338 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.111309 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tls-certs\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.162922 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.162881 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:01:07.212487 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.212446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-model-cache\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.212684 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.212503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tmp-dir\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.212684 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.212534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tls-certs\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.212684 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.212591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-home\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.212684 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.212638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-dshm\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.212684 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.212669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zt9l\" (UniqueName: \"kubernetes.io/projected/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-kube-api-access-9zt9l\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.212968 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.212895 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-model-cache\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.213090 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.213068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-home\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.213205 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.213162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tmp-dir\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.215130 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.215106 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-dshm\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.215256 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.215216 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tls-certs\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.221100 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.221078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zt9l\" (UniqueName: \"kubernetes.io/projected/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-kube-api-access-9zt9l\") pod \"sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.380546 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.380448 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:07.526694 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:07.526671 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l"] Jul 12 17:01:07.530959 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:01:07.530897 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ca45e42_fa92_4e27_85b3_3f3dd9d5e833.slice/crio-6ee4983f8d2f86fbe0e837ad5386ad0ddbbd62858054a3f16471c380be5ff3a7 WatchSource:0}: Error finding container 6ee4983f8d2f86fbe0e837ad5386ad0ddbbd62858054a3f16471c380be5ff3a7: Status 404 returned error can't find the container with id 6ee4983f8d2f86fbe0e837ad5386ad0ddbbd62858054a3f16471c380be5ff3a7 Jul 12 17:01:08.044809 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:08.044778 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" event={"ID":"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833","Type":"ContainerStarted","Data":"e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459"} Jul 12 17:01:08.044809 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:08.044820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" event={"ID":"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833","Type":"ContainerStarted","Data":"6ee4983f8d2f86fbe0e837ad5386ad0ddbbd62858054a3f16471c380be5ff3a7"} Jul 12 17:01:08.063922 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:08.063874 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" podStartSLOduration=1.063857772 podStartE2EDuration="1.063857772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:01:08.06149192 +0000 UTC m=+2166.830919828" watchObservedRunningTime="2026-07-12 17:01:08.063857772 +0000 UTC m=+2166.833285666" Jul 12 17:01:14.371831 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.371763 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" podUID="158e75df-d0d4-4538-ab68-bb56ff3dd2da" containerName="main" containerID="cri-o://fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4" gracePeriod=45 Jul 12 17:01:14.633040 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.632985 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 17:01:14.686997 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.686961 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-dshm\") pod \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " Jul 12 17:01:14.687142 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687068 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tmp-dir\") pod \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " Jul 12 17:01:14.687142 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687085 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-model-cache\") pod \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " Jul 12 17:01:14.687142 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-home\") pod \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " Jul 12 17:01:14.687295 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tls-certs\") pod \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " Jul 12 17:01:14.687295 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687167 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqjtg\" (UniqueName: \"kubernetes.io/projected/158e75df-d0d4-4538-ab68-bb56ff3dd2da-kube-api-access-dqjtg\") pod \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\" (UID: \"158e75df-d0d4-4538-ab68-bb56ff3dd2da\") " Jul 12 17:01:14.687423 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687369 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "158e75df-d0d4-4538-ab68-bb56ff3dd2da" (UID: "158e75df-d0d4-4538-ab68-bb56ff3dd2da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:01:14.687527 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687431 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-home" (OuterVolumeSpecName: "home") pod "158e75df-d0d4-4538-ab68-bb56ff3dd2da" (UID: "158e75df-d0d4-4538-ab68-bb56ff3dd2da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:01:14.687527 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687388 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-model-cache" (OuterVolumeSpecName: "model-cache") pod "158e75df-d0d4-4538-ab68-bb56ff3dd2da" (UID: "158e75df-d0d4-4538-ab68-bb56ff3dd2da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:01:14.687527 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687514 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tmp-dir\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:14.687652 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687530 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-model-cache\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:14.687652 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.687537 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-home\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:14.689290 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.689270 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-dshm" (OuterVolumeSpecName: "dshm") pod "158e75df-d0d4-4538-ab68-bb56ff3dd2da" (UID: "158e75df-d0d4-4538-ab68-bb56ff3dd2da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:01:14.689353 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.689283 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "158e75df-d0d4-4538-ab68-bb56ff3dd2da" (UID: "158e75df-d0d4-4538-ab68-bb56ff3dd2da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:01:14.689416 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.689398 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158e75df-d0d4-4538-ab68-bb56ff3dd2da-kube-api-access-dqjtg" (OuterVolumeSpecName: "kube-api-access-dqjtg") pod "158e75df-d0d4-4538-ab68-bb56ff3dd2da" (UID: "158e75df-d0d4-4538-ab68-bb56ff3dd2da"). InnerVolumeSpecName "kube-api-access-dqjtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:01:14.788230 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.788176 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158e75df-d0d4-4538-ab68-bb56ff3dd2da-tls-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:14.788230 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.788224 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqjtg\" (UniqueName: \"kubernetes.io/projected/158e75df-d0d4-4538-ab68-bb56ff3dd2da-kube-api-access-dqjtg\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:14.788230 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:14.788234 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158e75df-d0d4-4538-ab68-bb56ff3dd2da-dshm\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:15.087666 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.087633 2575 generic.go:356] "Generic (PLEG): container finished" podID="158e75df-d0d4-4538-ab68-bb56ff3dd2da" containerID="fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4" exitCode=0 Jul 12 17:01:15.087835 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.087677 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" Jul 12 17:01:15.087835 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.087718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" event={"ID":"158e75df-d0d4-4538-ab68-bb56ff3dd2da","Type":"ContainerDied","Data":"fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4"} Jul 12 17:01:15.087835 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.087762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l" event={"ID":"158e75df-d0d4-4538-ab68-bb56ff3dd2da","Type":"ContainerDied","Data":"dbe6ff3fdb14bdafd537d05d0d761ced3103891b90af7986fb28145d2b1fd07a"} Jul 12 17:01:15.087835 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.087780 2575 scope.go:117] "RemoveContainer" containerID="fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4" Jul 12 17:01:15.097215 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.097177 2575 scope.go:117] "RemoveContainer" containerID="fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4" Jul 12 17:01:15.097491 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:01:15.097473 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4\": container with ID starting with fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4 not found: ID does not exist" containerID="fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4" Jul 12 17:01:15.097550 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.097499 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4"} err="failed to get container status \"fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4\": rpc error: code = NotFound desc = could not find container \"fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4\": container with ID starting with fe3db130c26cb415fb633c25beeb4f89acab1ba03119538e5e48d7efbdeb64c4 not found: ID does not exist" Jul 12 17:01:15.108858 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.108809 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l"] Jul 12 17:01:15.112525 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.112504 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/auth-test-model-17d27e-kserve-85bc658c5-9nv8l"] Jul 12 17:01:15.287376 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.287338 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" podUID="e5736868-a6ae-4478-a140-3d1801b44350" containerName="main" containerID="cri-o://5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3" gracePeriod=45 Jul 12 17:01:15.541158 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.541132 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 17:01:15.594433 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594400 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-tmp-dir\") pod \"e5736868-a6ae-4478-a140-3d1801b44350\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " Jul 12 17:01:15.594603 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594438 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-home\") pod \"e5736868-a6ae-4478-a140-3d1801b44350\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " Jul 12 17:01:15.594603 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594578 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5736868-a6ae-4478-a140-3d1801b44350-tls-certs\") pod \"e5736868-a6ae-4478-a140-3d1801b44350\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " Jul 12 17:01:15.594683 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594638 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-home" (OuterVolumeSpecName: "home") pod "e5736868-a6ae-4478-a140-3d1801b44350" (UID: "e5736868-a6ae-4478-a140-3d1801b44350"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:01:15.594683 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594647 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-model-cache\") pod \"e5736868-a6ae-4478-a140-3d1801b44350\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " Jul 12 17:01:15.594761 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594677 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5736868-a6ae-4478-a140-3d1801b44350" (UID: "e5736868-a6ae-4478-a140-3d1801b44350"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:01:15.594761 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594695 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-dshm\") pod \"e5736868-a6ae-4478-a140-3d1801b44350\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " Jul 12 17:01:15.594842 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594807 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psmbq\" (UniqueName: \"kubernetes.io/projected/e5736868-a6ae-4478-a140-3d1801b44350-kube-api-access-psmbq\") pod \"e5736868-a6ae-4478-a140-3d1801b44350\" (UID: \"e5736868-a6ae-4478-a140-3d1801b44350\") " Jul 12 17:01:15.594842 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.594831 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5736868-a6ae-4478-a140-3d1801b44350" (UID: "e5736868-a6ae-4478-a140-3d1801b44350"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:01:15.595163 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.595139 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-tmp-dir\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:15.595163 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.595167 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-home\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:15.595360 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.595182 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-model-cache\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:15.596796 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.596775 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-dshm" (OuterVolumeSpecName: "dshm") pod "e5736868-a6ae-4478-a140-3d1801b44350" (UID: "e5736868-a6ae-4478-a140-3d1801b44350"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:01:15.596881 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.596857 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5736868-a6ae-4478-a140-3d1801b44350-kube-api-access-psmbq" (OuterVolumeSpecName: "kube-api-access-psmbq") pod "e5736868-a6ae-4478-a140-3d1801b44350" (UID: "e5736868-a6ae-4478-a140-3d1801b44350"). InnerVolumeSpecName "kube-api-access-psmbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:01:15.596927 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.596906 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5736868-a6ae-4478-a140-3d1801b44350-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5736868-a6ae-4478-a140-3d1801b44350" (UID: "e5736868-a6ae-4478-a140-3d1801b44350"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:01:15.696562 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.696529 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5736868-a6ae-4478-a140-3d1801b44350-tls-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:15.696562 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.696561 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5736868-a6ae-4478-a140-3d1801b44350-dshm\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:15.696805 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.696571 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psmbq\" (UniqueName: \"kubernetes.io/projected/e5736868-a6ae-4478-a140-3d1801b44350-kube-api-access-psmbq\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:01:15.875684 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:15.875650 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="158e75df-d0d4-4538-ab68-bb56ff3dd2da" path="/var/lib/kubelet/pods/158e75df-d0d4-4538-ab68-bb56ff3dd2da/volumes" Jul 12 17:01:16.095478 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.095392 2575 generic.go:356] "Generic (PLEG): container finished" podID="e5736868-a6ae-4478-a140-3d1801b44350" containerID="5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3" exitCode=0 Jul 12 17:01:16.095478 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.095434 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" Jul 12 17:01:16.095738 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.095474 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" event={"ID":"e5736868-a6ae-4478-a140-3d1801b44350","Type":"ContainerDied","Data":"5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3"} Jul 12 17:01:16.095738 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.095516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj" event={"ID":"e5736868-a6ae-4478-a140-3d1801b44350","Type":"ContainerDied","Data":"d2644a6bef09dd157bd373f7a740d9ae1ff16541bc7a3ec7cf9d15e29870040d"} Jul 12 17:01:16.095738 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.095532 2575 scope.go:117] "RemoveContainer" containerID="5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3" Jul 12 17:01:16.104311 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.104289 2575 scope.go:117] "RemoveContainer" containerID="5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3" Jul 12 17:01:16.104559 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:01:16.104542 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3\": container with ID starting with 5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3 not found: ID does not exist" containerID="5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3" Jul 12 17:01:16.104615 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.104567 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3"} err="failed to get container status \"5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3\": rpc error: code = NotFound desc = could not find container \"5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3\": container with ID starting with 5ea5683a0a3ac04f166fb547cb9927244349449970958e082465bf8611a3ebe3 not found: ID does not exist" Jul 12 17:01:16.112886 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.112861 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj"] Jul 12 17:01:16.115744 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:16.115721 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/auth-test-model-cc457c-kserve-dd4b7cc47-bqrzj"] Jul 12 17:01:17.381039 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:17.381004 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:17.381468 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:17.381051 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:17.393993 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:17.393971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:17.874828 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:17.874790 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5736868-a6ae-4478-a140-3d1801b44350" path="/var/lib/kubelet/pods/e5736868-a6ae-4478-a140-3d1801b44350/volumes" Jul 12 17:01:18.118962 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:18.118932 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:01:20.470054 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:20.470016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:01:28.270164 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.270122 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4"] Jul 12 17:01:28.270726 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.270706 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5736868-a6ae-4478-a140-3d1801b44350" containerName="main" Jul 12 17:01:28.270816 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.270730 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5736868-a6ae-4478-a140-3d1801b44350" containerName="main" Jul 12 17:01:28.270816 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.270752 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158e75df-d0d4-4538-ab68-bb56ff3dd2da" containerName="main" Jul 12 17:01:28.270816 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.270762 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="158e75df-d0d4-4538-ab68-bb56ff3dd2da" containerName="main" Jul 12 17:01:28.270960 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.270869 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="158e75df-d0d4-4538-ab68-bb56ff3dd2da" containerName="main" Jul 12 17:01:28.270960 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.270885 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5736868-a6ae-4478-a140-3d1801b44350" containerName="main" Jul 12 17:01:28.274268 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.274249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.277971 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.277949 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"openshift-service-ca.crt\"" Jul 12 17:01:28.278089 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.277982 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"kube-root-ca.crt\"" Jul 12 17:01:28.278089 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.277949 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"sub-test-model-cc457c-kserve-self-signed-certs\"" Jul 12 17:01:28.278089 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.278047 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"default-dockercfg-m4v8t\"" Jul 12 17:01:28.284221 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.284199 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4"] Jul 12 17:01:28.297798 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.297770 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:01:28.419210 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.419148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-tmp-dir\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.419393 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.419218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a177480c-8f16-4baa-849a-58caa690a817-tls-certs\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.419393 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.419274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdnn\" (UniqueName: \"kubernetes.io/projected/a177480c-8f16-4baa-849a-58caa690a817-kube-api-access-9hdnn\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.419393 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.419328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-home\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.419520 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.419412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-dshm\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.419520 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.419430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-model-cache\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.520602 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.520504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-dshm\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.520602 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.520549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-model-cache\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.520602 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.520585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-tmp-dir\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.520602 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.520606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a177480c-8f16-4baa-849a-58caa690a817-tls-certs\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.520930 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.520626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdnn\" (UniqueName: \"kubernetes.io/projected/a177480c-8f16-4baa-849a-58caa690a817-kube-api-access-9hdnn\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.520930 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.520660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-home\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.521043 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.521003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-model-cache\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.521099 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.521055 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-tmp-dir\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.521099 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.521073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-home\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.522932 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.522905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-dshm\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.523214 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.523179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a177480c-8f16-4baa-849a-58caa690a817-tls-certs\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.528676 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.528651 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdnn\" (UniqueName: \"kubernetes.io/projected/a177480c-8f16-4baa-849a-58caa690a817-kube-api-access-9hdnn\") pod \"sub-test-model-cc457c-kserve-874d6b69-r2fg4\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.584776 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.584735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:28.725998 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:28.725970 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4"] Jul 12 17:01:28.727489 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:01:28.727464 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda177480c_8f16_4baa_849a_58caa690a817.slice/crio-adc3f02c54c55fbe98e831630916c35796be3aa10d093b99a8c9d509eb0a79f4 WatchSource:0}: Error finding container adc3f02c54c55fbe98e831630916c35796be3aa10d093b99a8c9d509eb0a79f4: Status 404 returned error can't find the container with id adc3f02c54c55fbe98e831630916c35796be3aa10d093b99a8c9d509eb0a79f4 Jul 12 17:01:29.176714 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:29.176673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" event={"ID":"a177480c-8f16-4baa-849a-58caa690a817","Type":"ContainerStarted","Data":"f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51"} Jul 12 17:01:29.176714 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:29.176721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" event={"ID":"a177480c-8f16-4baa-849a-58caa690a817","Type":"ContainerStarted","Data":"adc3f02c54c55fbe98e831630916c35796be3aa10d093b99a8c9d509eb0a79f4"} Jul 12 17:01:29.194100 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:29.194048 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" podStartSLOduration=1.1940316850000001 podStartE2EDuration="1.194031685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:01:29.19304855 +0000 UTC m=+2187.962476446" watchObservedRunningTime="2026-07-12 17:01:29.194031685 +0000 UTC m=+2187.963459579" Jul 12 17:01:31.943594 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:31.943560 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:01:32.044991 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:32.044953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:01:33.674799 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:33.674752 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:01:38.585131 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:38.585088 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:38.585640 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:38.585144 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:38.597722 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:38.597694 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:39.247940 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:39.247911 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:01:40.912059 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:40.912015 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:01:48.971565 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:01:48.971526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:02:03.746251 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:03.746216 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l"] Jul 12 17:02:03.768177 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:03.768145 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:02:04.681246 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:04.681204 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4"] Jul 12 17:02:04.693332 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:04.693299 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:02:09.362917 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:09.362884 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:02:18.766882 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:18.766838 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" podUID="3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" containerName="main" containerID="cri-o://e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459" gracePeriod=45 Jul 12 17:02:19.013314 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.013286 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:02:19.199223 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.199178 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zt9l\" (UniqueName: \"kubernetes.io/projected/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-kube-api-access-9zt9l\") pod \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " Jul 12 17:02:19.199414 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.199246 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tmp-dir\") pod \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " Jul 12 17:02:19.199414 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.199282 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-dshm\") pod \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " Jul 12 17:02:19.199414 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.199362 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tls-certs\") pod \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " Jul 12 17:02:19.199579 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.199461 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-home\") pod \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " Jul 12 17:02:19.199579 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.199498 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-model-cache\") pod \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\" (UID: \"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833\") " Jul 12 17:02:19.199878 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.199795 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" (UID: "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:02:19.199878 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.199801 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-home" (OuterVolumeSpecName: "home") pod "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" (UID: "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:02:19.200246 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.200214 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" (UID: "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:02:19.201481 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.201454 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-kube-api-access-9zt9l" (OuterVolumeSpecName: "kube-api-access-9zt9l") pod "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" (UID: "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833"). InnerVolumeSpecName "kube-api-access-9zt9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:02:19.201705 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.201683 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-dshm" (OuterVolumeSpecName: "dshm") pod "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" (UID: "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:02:19.201811 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.201790 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" (UID: "3ca45e42-fa92-4e27-85b3-3f3dd9d5e833"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:02:19.301109 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.301076 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-model-cache\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:19.301109 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.301105 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zt9l\" (UniqueName: \"kubernetes.io/projected/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-kube-api-access-9zt9l\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:19.301109 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.301114 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tmp-dir\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:19.301381 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.301125 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-dshm\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:19.301381 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.301134 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-tls-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:19.301381 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.301141 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833-home\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:19.478005 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.477915 2575 generic.go:356] "Generic (PLEG): container finished" podID="3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" containerID="e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459" exitCode=0 Jul 12 17:02:19.478005 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.477956 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" Jul 12 17:02:19.478180 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.478008 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" event={"ID":"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833","Type":"ContainerDied","Data":"e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459"} Jul 12 17:02:19.478180 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.478046 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l" event={"ID":"3ca45e42-fa92-4e27-85b3-3f3dd9d5e833","Type":"ContainerDied","Data":"6ee4983f8d2f86fbe0e837ad5386ad0ddbbd62858054a3f16471c380be5ff3a7"} Jul 12 17:02:19.478180 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.478062 2575 scope.go:117] "RemoveContainer" containerID="e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459" Jul 12 17:02:19.488626 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.488610 2575 scope.go:117] "RemoveContainer" containerID="e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459" Jul 12 17:02:19.488884 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:02:19.488868 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459\": container with ID starting with e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459 not found: ID does not exist" containerID="e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459" Jul 12 17:02:19.488934 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.488891 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459"} err="failed to get container status \"e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459\": rpc error: code = NotFound desc = could not find container \"e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459\": container with ID starting with e42e4b9f13ca06dfa3f0b9c2104faae9788a350f2cb95fc94e844da093074459 not found: ID does not exist" Jul 12 17:02:19.500729 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.500706 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l"] Jul 12 17:02:19.507179 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.507159 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/sub-test-model-17d27e-kserve-7b4b865c5f-l9l7l"] Jul 12 17:02:19.701828 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.701768 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" podUID="a177480c-8f16-4baa-849a-58caa690a817" containerName="main" containerID="cri-o://f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51" gracePeriod=45 Jul 12 17:02:19.877330 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.877299 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" path="/var/lib/kubelet/pods/3ca45e42-fa92-4e27-85b3-3f3dd9d5e833/volumes" Jul 12 17:02:19.960786 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:19.960762 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:02:20.109550 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.109456 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hdnn\" (UniqueName: \"kubernetes.io/projected/a177480c-8f16-4baa-849a-58caa690a817-kube-api-access-9hdnn\") pod \"a177480c-8f16-4baa-849a-58caa690a817\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " Jul 12 17:02:20.109550 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.109532 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-dshm\") pod \"a177480c-8f16-4baa-849a-58caa690a817\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " Jul 12 17:02:20.109781 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.109568 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-model-cache\") pod \"a177480c-8f16-4baa-849a-58caa690a817\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " Jul 12 17:02:20.109781 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.109595 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a177480c-8f16-4baa-849a-58caa690a817-tls-certs\") pod \"a177480c-8f16-4baa-849a-58caa690a817\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " Jul 12 17:02:20.109781 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.109638 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-tmp-dir\") pod \"a177480c-8f16-4baa-849a-58caa690a817\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " Jul 12 17:02:20.109781 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.109670 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-home\") pod \"a177480c-8f16-4baa-849a-58caa690a817\" (UID: \"a177480c-8f16-4baa-849a-58caa690a817\") " Jul 12 17:02:20.110001 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.109845 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-model-cache" (OuterVolumeSpecName: "model-cache") pod "a177480c-8f16-4baa-849a-58caa690a817" (UID: "a177480c-8f16-4baa-849a-58caa690a817"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:02:20.110086 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.110066 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-home" (OuterVolumeSpecName: "home") pod "a177480c-8f16-4baa-849a-58caa690a817" (UID: "a177480c-8f16-4baa-849a-58caa690a817"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:02:20.110166 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.110056 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-model-cache\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:20.110269 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.110231 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a177480c-8f16-4baa-849a-58caa690a817" (UID: "a177480c-8f16-4baa-849a-58caa690a817"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:02:20.112094 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.112061 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-dshm" (OuterVolumeSpecName: "dshm") pod "a177480c-8f16-4baa-849a-58caa690a817" (UID: "a177480c-8f16-4baa-849a-58caa690a817"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:02:20.113614 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.113558 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a177480c-8f16-4baa-849a-58caa690a817-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a177480c-8f16-4baa-849a-58caa690a817" (UID: "a177480c-8f16-4baa-849a-58caa690a817"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:02:20.113728 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.113668 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a177480c-8f16-4baa-849a-58caa690a817-kube-api-access-9hdnn" (OuterVolumeSpecName: "kube-api-access-9hdnn") pod "a177480c-8f16-4baa-849a-58caa690a817" (UID: "a177480c-8f16-4baa-849a-58caa690a817"). InnerVolumeSpecName "kube-api-access-9hdnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:02:20.211383 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.211345 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-tmp-dir\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:20.211383 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.211374 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-home\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:20.211383 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.211383 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdnn\" (UniqueName: \"kubernetes.io/projected/a177480c-8f16-4baa-849a-58caa690a817-kube-api-access-9hdnn\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:20.211383 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.211391 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a177480c-8f16-4baa-849a-58caa690a817-dshm\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:20.211383 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.211399 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a177480c-8f16-4baa-849a-58caa690a817-tls-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:02:20.485262 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.485228 2575 generic.go:356] "Generic (PLEG): container finished" podID="a177480c-8f16-4baa-849a-58caa690a817" containerID="f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51" exitCode=0 Jul 12 17:02:20.485433 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.485273 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" Jul 12 17:02:20.485433 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.485314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" event={"ID":"a177480c-8f16-4baa-849a-58caa690a817","Type":"ContainerDied","Data":"f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51"} Jul 12 17:02:20.485433 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.485354 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4" event={"ID":"a177480c-8f16-4baa-849a-58caa690a817","Type":"ContainerDied","Data":"adc3f02c54c55fbe98e831630916c35796be3aa10d093b99a8c9d509eb0a79f4"} Jul 12 17:02:20.485433 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.485371 2575 scope.go:117] "RemoveContainer" containerID="f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51" Jul 12 17:02:20.494533 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.494518 2575 scope.go:117] "RemoveContainer" containerID="f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51" Jul 12 17:02:20.494779 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:02:20.494765 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51\": container with ID starting with f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51 not found: ID does not exist" containerID="f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51" Jul 12 17:02:20.494818 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.494786 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51"} err="failed to get container status \"f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51\": rpc error: code = NotFound desc = could not find container \"f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51\": container with ID starting with f54f48e0aa5bf507a26b699e19c46a01dff285200d9d9be377d6bb137b88ee51 not found: ID does not exist" Jul 12 17:02:20.506668 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.506517 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4"] Jul 12 17:02:20.510865 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:20.510844 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/sub-test-model-cc457c-kserve-874d6b69-r2fg4"] Jul 12 17:02:21.876110 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:21.876077 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a177480c-8f16-4baa-849a-58caa690a817" path="/var/lib/kubelet/pods/a177480c-8f16-4baa-849a-58caa690a817/volumes" Jul 12 17:02:22.470060 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.470030 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n"] Jul 12 17:02:22.470578 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.470557 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" containerName="main" Jul 12 17:02:22.470578 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.470580 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" containerName="main" Jul 12 17:02:22.470744 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.470599 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a177480c-8f16-4baa-849a-58caa690a817" containerName="main" Jul 12 17:02:22.470744 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.470607 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a177480c-8f16-4baa-849a-58caa690a817" containerName="main" Jul 12 17:02:22.470744 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.470724 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca45e42-fa92-4e27-85b3-3f3dd9d5e833" containerName="main" Jul 12 17:02:22.470744 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.470737 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a177480c-8f16-4baa-849a-58caa690a817" containerName="main" Jul 12 17:02:22.475058 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.475038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.478916 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.478895 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"kube-root-ca.crt\"" Jul 12 17:02:22.479045 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.478927 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"rate-test-model-17d27e-kserve-self-signed-certs\"" Jul 12 17:02:22.479045 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.478975 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"default-dockercfg-ljksn\"" Jul 12 17:02:22.479045 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.478995 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-17d27e\"/\"openshift-service-ca.crt\"" Jul 12 17:02:22.483872 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.483849 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n"] Jul 12 17:02:22.492394 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.492362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:02:22.632395 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.632356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5vjs\" (UniqueName: \"kubernetes.io/projected/60e78df5-1cd4-4a24-afea-5ad3b566486d-kube-api-access-j5vjs\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.632641 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.632614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-model-cache\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.632849 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.632659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-dshm\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.632849 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.632697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e78df5-1cd4-4a24-afea-5ad3b566486d-tls-certs\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.632849 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.632728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-tmp-dir\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.632849 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.632773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-home\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.733592 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.733501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5vjs\" (UniqueName: \"kubernetes.io/projected/60e78df5-1cd4-4a24-afea-5ad3b566486d-kube-api-access-j5vjs\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.733592 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.733549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-model-cache\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.733592 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.733572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-dshm\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.733592 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.733596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e78df5-1cd4-4a24-afea-5ad3b566486d-tls-certs\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.733859 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.733616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-tmp-dir\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.733859 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.733642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-home\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.734012 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.733991 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-model-cache\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.734081 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.734054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-home\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.734137 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.734093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-tmp-dir\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.735813 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.735792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-dshm\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.736115 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.736101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e78df5-1cd4-4a24-afea-5ad3b566486d-tls-certs\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.741463 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.741436 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5vjs\" (UniqueName: \"kubernetes.io/projected/60e78df5-1cd4-4a24-afea-5ad3b566486d-kube-api-access-j5vjs\") pod \"rate-test-model-17d27e-kserve-655b97fd75-jvn2n\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.784991 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.784960 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:22.920990 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:22.920959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n"] Jul 12 17:02:22.922644 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:02:22.922620 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60e78df5_1cd4_4a24_afea_5ad3b566486d.slice/crio-ec57bd4d1c6476543ec054ed9a1c153466a49c488f0b65a2dc0495e6d2fbf9f4 WatchSource:0}: Error finding container ec57bd4d1c6476543ec054ed9a1c153466a49c488f0b65a2dc0495e6d2fbf9f4: Status 404 returned error can't find the container with id ec57bd4d1c6476543ec054ed9a1c153466a49c488f0b65a2dc0495e6d2fbf9f4 Jul 12 17:02:23.505882 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:23.505808 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" event={"ID":"60e78df5-1cd4-4a24-afea-5ad3b566486d","Type":"ContainerStarted","Data":"9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5"} Jul 12 17:02:23.505882 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:23.505848 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" event={"ID":"60e78df5-1cd4-4a24-afea-5ad3b566486d","Type":"ContainerStarted","Data":"ec57bd4d1c6476543ec054ed9a1c153466a49c488f0b65a2dc0495e6d2fbf9f4"} Jul 12 17:02:23.525748 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:23.525686 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" podStartSLOduration=1.5256697689999998 podStartE2EDuration="1.525669769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:02:23.523977751 +0000 UTC m=+2242.293405665" watchObservedRunningTime="2026-07-12 17:02:23.525669769 +0000 UTC m=+2242.295097665" Jul 12 17:02:26.274977 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:26.274942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:02:32.785205 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:32.785167 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:32.785684 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:32.785236 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:32.797837 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:32.797812 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:33.575918 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:33.575887 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:02:38.365236 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.365200 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v"] Jul 12 17:02:38.368719 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.368702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.372153 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.372123 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"openshift-service-ca.crt\"" Jul 12 17:02:38.372153 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.372155 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"kube-root-ca.crt\"" Jul 12 17:02:38.372355 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.372180 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"rate-test-model-cc457c-kserve-self-signed-certs\"" Jul 12 17:02:38.372355 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.372153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cc457c\"/\"default-dockercfg-m4v8t\"" Jul 12 17:02:38.377851 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.377831 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v"] Jul 12 17:02:38.468526 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.468492 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:02:38.470579 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.470557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81541e68-0fa3-4ce2-834d-8dd441c259ca-tls-certs\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.470698 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.470589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-model-cache\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.470698 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.470621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-dshm\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.470698 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.470680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zwf\" (UniqueName: \"kubernetes.io/projected/81541e68-0fa3-4ce2-834d-8dd441c259ca-kube-api-access-h5zwf\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.470798 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.470721 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-tmp-dir\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.470798 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.470737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-home\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.571520 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.571480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-dshm\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.571697 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.571572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zwf\" (UniqueName: \"kubernetes.io/projected/81541e68-0fa3-4ce2-834d-8dd441c259ca-kube-api-access-h5zwf\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.571697 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.571626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-tmp-dir\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.571697 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.571651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-home\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.571878 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.571703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81541e68-0fa3-4ce2-834d-8dd441c259ca-tls-certs\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.571878 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.571736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-model-cache\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.572086 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.572059 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-tmp-dir\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.572158 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.572095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-model-cache\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.572158 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.572062 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-home\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.573965 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.573942 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-dshm\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.574227 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.574211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81541e68-0fa3-4ce2-834d-8dd441c259ca-tls-certs\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.579447 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.579420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zwf\" (UniqueName: \"kubernetes.io/projected/81541e68-0fa3-4ce2-834d-8dd441c259ca-kube-api-access-h5zwf\") pod \"rate-test-model-cc457c-kserve-64d777958c-z5w9v\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.678735 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.678704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:38.821488 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:38.821461 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v"] Jul 12 17:02:38.822494 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:02:38.822472 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81541e68_0fa3_4ce2_834d_8dd441c259ca.slice/crio-a18be02cc3cac5dd6d2f0e737c0babcb9320dcf207972875fe45746dd96b955c WatchSource:0}: Error finding container a18be02cc3cac5dd6d2f0e737c0babcb9320dcf207972875fe45746dd96b955c: Status 404 returned error can't find the container with id a18be02cc3cac5dd6d2f0e737c0babcb9320dcf207972875fe45746dd96b955c Jul 12 17:02:39.601950 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:39.601917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" event={"ID":"81541e68-0fa3-4ce2-834d-8dd441c259ca","Type":"ContainerStarted","Data":"584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19"} Jul 12 17:02:39.602326 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:39.601958 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" event={"ID":"81541e68-0fa3-4ce2-834d-8dd441c259ca","Type":"ContainerStarted","Data":"a18be02cc3cac5dd6d2f0e737c0babcb9320dcf207972875fe45746dd96b955c"} Jul 12 17:02:39.619389 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:39.619333 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" podStartSLOduration=1.6193145310000001 podStartE2EDuration="1.619314531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 17:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:02:39.619063834 +0000 UTC m=+2258.388491728" watchObservedRunningTime="2026-07-12 17:02:39.619314531 +0000 UTC m=+2258.388742426" Jul 12 17:02:45.774100 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:45.774066 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:02:48.678875 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:48.678834 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:48.678875 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:48.678887 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:48.692284 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:48.692257 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:49.670877 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:49.670851 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:02:53.152395 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:53.152349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:02:53.258391 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:02:53.258358 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:05.766679 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:05.766645 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:03:21.972897 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:21.972861 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:03:22.846075 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:22.846040 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:22.947238 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:22.947200 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:28.112505 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:28.112473 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n"] Jul 12 17:03:28.121878 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:28.121846 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:29.054626 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:29.054592 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v"] Jul 12 17:03:29.071762 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:29.071728 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:30.013547 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.013516 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7387-140309746834480"] Jul 12 17:03:30.016752 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.016734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309746834480" Jul 12 17:03:30.019201 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.019164 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-vkqq6\"" Jul 12 17:03:30.022526 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.022502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309746834480"] Jul 12 17:03:30.035125 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.035096 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:30.060373 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.060338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdd8m\" (UniqueName: \"kubernetes.io/projected/dbff6c30-5864-49fc-976b-1797ebd92dcc-kube-api-access-sdd8m\") pod \"test-curl-7387-140309746834480\" (UID: \"dbff6c30-5864-49fc-976b-1797ebd92dcc\") " pod="openshift-ingress/test-curl-7387-140309746834480" Jul 12 17:03:30.136097 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.136064 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:30.161968 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.161935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdd8m\" (UniqueName: \"kubernetes.io/projected/dbff6c30-5864-49fc-976b-1797ebd92dcc-kube-api-access-sdd8m\") pod \"test-curl-7387-140309746834480\" (UID: \"dbff6c30-5864-49fc-976b-1797ebd92dcc\") " pod="openshift-ingress/test-curl-7387-140309746834480" Jul 12 17:03:30.170257 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.170226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdd8m\" (UniqueName: \"kubernetes.io/projected/dbff6c30-5864-49fc-976b-1797ebd92dcc-kube-api-access-sdd8m\") pod \"test-curl-7387-140309746834480\" (UID: \"dbff6c30-5864-49fc-976b-1797ebd92dcc\") " pod="openshift-ingress/test-curl-7387-140309746834480" Jul 12 17:03:30.326919 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.326823 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309746834480" Jul 12 17:03:30.459337 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.459311 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309746834480"] Jul 12 17:03:30.461167 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:03:30.461142 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbff6c30_5864_49fc_976b_1797ebd92dcc.slice/crio-085599a741e812ef4ec2b07c72836611a4681dd411959f2c57ff4798602b3204 WatchSource:0}: Error finding container 085599a741e812ef4ec2b07c72836611a4681dd411959f2c57ff4798602b3204: Status 404 returned error can't find the container with id 085599a741e812ef4ec2b07c72836611a4681dd411959f2c57ff4798602b3204 Jul 12 17:03:30.462533 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.462516 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 17:03:30.891496 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:30.891460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309746834480" event={"ID":"dbff6c30-5864-49fc-976b-1797ebd92dcc","Type":"ContainerStarted","Data":"085599a741e812ef4ec2b07c72836611a4681dd411959f2c57ff4798602b3204"} Jul 12 17:03:31.898633 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:31.898601 2575 generic.go:356] "Generic (PLEG): container finished" podID="dbff6c30-5864-49fc-976b-1797ebd92dcc" containerID="c12e563942bb8890a1e0cc4fe21e38a13630a9e75ab0c24c96ef74631ab87cda" exitCode=0 Jul 12 17:03:31.898999 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:31.898655 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309746834480" event={"ID":"dbff6c30-5864-49fc-976b-1797ebd92dcc","Type":"ContainerDied","Data":"c12e563942bb8890a1e0cc4fe21e38a13630a9e75ab0c24c96ef74631ab87cda"} Jul 12 17:03:33.051817 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.051795 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309746834480" Jul 12 17:03:33.071618 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.071594 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7387-140309746834480_dbff6c30-5864-49fc-976b-1797ebd92dcc/test-curl-7387-140309746834480/0.log" Jul 12 17:03:33.091240 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.091211 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdd8m\" (UniqueName: \"kubernetes.io/projected/dbff6c30-5864-49fc-976b-1797ebd92dcc-kube-api-access-sdd8m\") pod \"dbff6c30-5864-49fc-976b-1797ebd92dcc\" (UID: \"dbff6c30-5864-49fc-976b-1797ebd92dcc\") " Jul 12 17:03:33.092958 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.092937 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7387-140309746834480"] Jul 12 17:03:33.093379 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.093355 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbff6c30-5864-49fc-976b-1797ebd92dcc-kube-api-access-sdd8m" (OuterVolumeSpecName: "kube-api-access-sdd8m") pod "dbff6c30-5864-49fc-976b-1797ebd92dcc" (UID: "dbff6c30-5864-49fc-976b-1797ebd92dcc"). InnerVolumeSpecName "kube-api-access-sdd8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:03:33.096160 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.096139 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7387-140309746834480"] Jul 12 17:03:33.192717 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.192690 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdd8m\" (UniqueName: \"kubernetes.io/projected/dbff6c30-5864-49fc-976b-1797ebd92dcc-kube-api-access-sdd8m\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:33.223267 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.223235 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7387-140309746834224"] Jul 12 17:03:33.223664 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.223650 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbff6c30-5864-49fc-976b-1797ebd92dcc" containerName="test-curl-7387-140309746834480" Jul 12 17:03:33.223707 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.223665 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbff6c30-5864-49fc-976b-1797ebd92dcc" containerName="test-curl-7387-140309746834480" Jul 12 17:03:33.223755 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.223746 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbff6c30-5864-49fc-976b-1797ebd92dcc" containerName="test-curl-7387-140309746834480" Jul 12 17:03:33.226738 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.226721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309746834224" Jul 12 17:03:33.230700 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.230672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309746834224"] Jul 12 17:03:33.294211 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.294151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82gm\" (UniqueName: \"kubernetes.io/projected/dac2216b-61fa-4289-86d3-0d61ca4a0807-kube-api-access-h82gm\") pod \"test-curl-7387-140309746834224\" (UID: \"dac2216b-61fa-4289-86d3-0d61ca4a0807\") " pod="openshift-ingress/test-curl-7387-140309746834224" Jul 12 17:03:33.394794 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.394755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82gm\" (UniqueName: \"kubernetes.io/projected/dac2216b-61fa-4289-86d3-0d61ca4a0807-kube-api-access-h82gm\") pod \"test-curl-7387-140309746834224\" (UID: \"dac2216b-61fa-4289-86d3-0d61ca4a0807\") " pod="openshift-ingress/test-curl-7387-140309746834224" Jul 12 17:03:33.403405 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.403368 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82gm\" (UniqueName: \"kubernetes.io/projected/dac2216b-61fa-4289-86d3-0d61ca4a0807-kube-api-access-h82gm\") pod \"test-curl-7387-140309746834224\" (UID: \"dac2216b-61fa-4289-86d3-0d61ca4a0807\") " pod="openshift-ingress/test-curl-7387-140309746834224" Jul 12 17:03:33.537595 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.537499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309746834224" Jul 12 17:03:33.681510 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.678466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309746834224"] Jul 12 17:03:33.876539 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.876458 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbff6c30-5864-49fc-976b-1797ebd92dcc" path="/var/lib/kubelet/pods/dbff6c30-5864-49fc-976b-1797ebd92dcc/volumes" Jul 12 17:03:33.910199 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.910169 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309746834480" Jul 12 17:03:33.910356 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.910219 2575 scope.go:117] "RemoveContainer" containerID="c12e563942bb8890a1e0cc4fe21e38a13630a9e75ab0c24c96ef74631ab87cda" Jul 12 17:03:33.911662 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:33.911541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309746834224" event={"ID":"dac2216b-61fa-4289-86d3-0d61ca4a0807","Type":"ContainerStarted","Data":"802cbdcecf8515844555a05b57fc04482a3ae03803706b26551fbdcd6e02f9ee"} Jul 12 17:03:34.919720 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:34.919682 2575 generic.go:356] "Generic (PLEG): container finished" podID="dac2216b-61fa-4289-86d3-0d61ca4a0807" containerID="c280b5bbd13cf17d961dca0ae3d36b099558ce5b00749dec1df0eb1dc8e46a08" exitCode=0 Jul 12 17:03:34.920125 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:34.919773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309746834224" event={"ID":"dac2216b-61fa-4289-86d3-0d61ca4a0807","Type":"ContainerDied","Data":"c280b5bbd13cf17d961dca0ae3d36b099558ce5b00749dec1df0eb1dc8e46a08"} Jul 12 17:03:36.068844 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.068822 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309746834224" Jul 12 17:03:36.089309 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.089281 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7387-140309746834224_dac2216b-61fa-4289-86d3-0d61ca4a0807/test-curl-7387-140309746834224/0.log" Jul 12 17:03:36.112518 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.112493 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7387-140309746834224"] Jul 12 17:03:36.116459 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.116437 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7387-140309746834224"] Jul 12 17:03:36.119080 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.119064 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h82gm\" (UniqueName: \"kubernetes.io/projected/dac2216b-61fa-4289-86d3-0d61ca4a0807-kube-api-access-h82gm\") pod \"dac2216b-61fa-4289-86d3-0d61ca4a0807\" (UID: \"dac2216b-61fa-4289-86d3-0d61ca4a0807\") " Jul 12 17:03:36.121082 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.121058 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac2216b-61fa-4289-86d3-0d61ca4a0807-kube-api-access-h82gm" (OuterVolumeSpecName: "kube-api-access-h82gm") pod "dac2216b-61fa-4289-86d3-0d61ca4a0807" (UID: "dac2216b-61fa-4289-86d3-0d61ca4a0807"). InnerVolumeSpecName "kube-api-access-h82gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:03:36.220969 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.220886 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82gm\" (UniqueName: \"kubernetes.io/projected/dac2216b-61fa-4289-86d3-0d61ca4a0807-kube-api-access-h82gm\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:36.269819 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.269784 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:36.270261 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.270242 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac2216b-61fa-4289-86d3-0d61ca4a0807" containerName="test-curl-7387-140309746834224" Jul 12 17:03:36.270341 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.270264 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac2216b-61fa-4289-86d3-0d61ca4a0807" containerName="test-curl-7387-140309746834224" Jul 12 17:03:36.270427 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.270414 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac2216b-61fa-4289-86d3-0d61ca4a0807" containerName="test-curl-7387-140309746834224" Jul 12 17:03:36.273441 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.273426 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:36.279250 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.279227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:36.321561 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.321536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshxv\" (UniqueName: \"kubernetes.io/projected/72d0b6fa-8f17-462c-a53f-d5f83b2880dd-kube-api-access-bshxv\") pod \"test-curl-7387-140309749413760\" (UID: \"72d0b6fa-8f17-462c-a53f-d5f83b2880dd\") " pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:36.422680 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.422644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshxv\" (UniqueName: \"kubernetes.io/projected/72d0b6fa-8f17-462c-a53f-d5f83b2880dd-kube-api-access-bshxv\") pod \"test-curl-7387-140309749413760\" (UID: \"72d0b6fa-8f17-462c-a53f-d5f83b2880dd\") " pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:36.430299 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.430274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshxv\" (UniqueName: \"kubernetes.io/projected/72d0b6fa-8f17-462c-a53f-d5f83b2880dd-kube-api-access-bshxv\") pod \"test-curl-7387-140309749413760\" (UID: \"72d0b6fa-8f17-462c-a53f-d5f83b2880dd\") " pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:36.584007 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.583927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:36.714885 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.714855 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:36.716227 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:03:36.716202 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d0b6fa_8f17_462c_a53f_d5f83b2880dd.slice/crio-6c7642247ff0007b22e1312b492d629965f81f6276f7d4c5f620dbe6b0bd07e9 WatchSource:0}: Error finding container 6c7642247ff0007b22e1312b492d629965f81f6276f7d4c5f620dbe6b0bd07e9: Status 404 returned error can't find the container with id 6c7642247ff0007b22e1312b492d629965f81f6276f7d4c5f620dbe6b0bd07e9 Jul 12 17:03:36.933454 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.933431 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309746834224" Jul 12 17:03:36.933454 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.933445 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="802cbdcecf8515844555a05b57fc04482a3ae03803706b26551fbdcd6e02f9ee" Jul 12 17:03:36.934654 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:36.934628 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309749413760" event={"ID":"72d0b6fa-8f17-462c-a53f-d5f83b2880dd","Type":"ContainerStarted","Data":"6c7642247ff0007b22e1312b492d629965f81f6276f7d4c5f620dbe6b0bd07e9"} Jul 12 17:03:36.944237 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:03:36.944176 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7387-140309746834224\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="dac2216b-61fa-4289-86d3-0d61ca4a0807" pod="openshift-ingress/test-curl-7387-140309746834224" Jul 12 17:03:37.880507 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:37.880475 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac2216b-61fa-4289-86d3-0d61ca4a0807" path="/var/lib/kubelet/pods/dac2216b-61fa-4289-86d3-0d61ca4a0807/volumes" Jul 12 17:03:37.942467 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:37.942432 2575 generic.go:356] "Generic (PLEG): container finished" podID="72d0b6fa-8f17-462c-a53f-d5f83b2880dd" containerID="6642f68feb4d3cae6d5f3fbeb6349f1a0cfa5b2784e06001c2ca1ddf777ac43f" exitCode=0 Jul 12 17:03:37.942622 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:37.942493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309749413760" event={"ID":"72d0b6fa-8f17-462c-a53f-d5f83b2880dd","Type":"ContainerDied","Data":"6642f68feb4d3cae6d5f3fbeb6349f1a0cfa5b2784e06001c2ca1ddf777ac43f"} Jul 12 17:03:39.096378 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.096355 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:39.115425 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.115402 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7387-140309749413760_72d0b6fa-8f17-462c-a53f-d5f83b2880dd/test-curl-7387-140309749413760/0.log" Jul 12 17:03:39.139202 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.139169 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:39.143162 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.143140 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:39.146821 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.146804 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bshxv\" (UniqueName: \"kubernetes.io/projected/72d0b6fa-8f17-462c-a53f-d5f83b2880dd-kube-api-access-bshxv\") pod \"72d0b6fa-8f17-462c-a53f-d5f83b2880dd\" (UID: \"72d0b6fa-8f17-462c-a53f-d5f83b2880dd\") " Jul 12 17:03:39.148758 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.148734 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d0b6fa-8f17-462c-a53f-d5f83b2880dd-kube-api-access-bshxv" (OuterVolumeSpecName: "kube-api-access-bshxv") pod "72d0b6fa-8f17-462c-a53f-d5f83b2880dd" (UID: "72d0b6fa-8f17-462c-a53f-d5f83b2880dd"). InnerVolumeSpecName "kube-api-access-bshxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:03:39.248261 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.248165 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bshxv\" (UniqueName: \"kubernetes.io/projected/72d0b6fa-8f17-462c-a53f-d5f83b2880dd-kube-api-access-bshxv\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:39.265210 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.263222 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:39.265210 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.263807 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d0b6fa-8f17-462c-a53f-d5f83b2880dd" containerName="test-curl-7387-140309749413760" Jul 12 17:03:39.265210 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.263824 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d0b6fa-8f17-462c-a53f-d5f83b2880dd" containerName="test-curl-7387-140309749413760" Jul 12 17:03:39.265210 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.263911 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d0b6fa-8f17-462c-a53f-d5f83b2880dd" containerName="test-curl-7387-140309749413760" Jul 12 17:03:39.279142 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.278537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:39.279369 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.278434 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:39.349255 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.349225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlvl\" (UniqueName: \"kubernetes.io/projected/6a71645c-fb28-4ade-b3cf-d4d16e3259fa-kube-api-access-qwlvl\") pod \"test-curl-7387-140309749413760\" (UID: \"6a71645c-fb28-4ade-b3cf-d4d16e3259fa\") " pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:39.450733 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.450690 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlvl\" (UniqueName: \"kubernetes.io/projected/6a71645c-fb28-4ade-b3cf-d4d16e3259fa-kube-api-access-qwlvl\") pod \"test-curl-7387-140309749413760\" (UID: \"6a71645c-fb28-4ade-b3cf-d4d16e3259fa\") " pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:39.458853 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.458830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlvl\" (UniqueName: \"kubernetes.io/projected/6a71645c-fb28-4ade-b3cf-d4d16e3259fa-kube-api-access-qwlvl\") pod \"test-curl-7387-140309749413760\" (UID: \"6a71645c-fb28-4ade-b3cf-d4d16e3259fa\") " pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:39.593859 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.593782 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:39.725300 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.725276 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:39.726479 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:03:39.726453 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a71645c_fb28_4ade_b3cf_d4d16e3259fa.slice/crio-a857355d75166e2a5d172b1ad59a364c723b387d1c315452aab42eb0dd233344 WatchSource:0}: Error finding container a857355d75166e2a5d172b1ad59a364c723b387d1c315452aab42eb0dd233344: Status 404 returned error can't find the container with id a857355d75166e2a5d172b1ad59a364c723b387d1c315452aab42eb0dd233344 Jul 12 17:03:39.876906 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.876829 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72d0b6fa-8f17-462c-a53f-d5f83b2880dd" path="/var/lib/kubelet/pods/72d0b6fa-8f17-462c-a53f-d5f83b2880dd/volumes" Jul 12 17:03:39.954702 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.954658 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309749413760" event={"ID":"6a71645c-fb28-4ade-b3cf-d4d16e3259fa","Type":"ContainerStarted","Data":"a857355d75166e2a5d172b1ad59a364c723b387d1c315452aab42eb0dd233344"} Jul 12 17:03:39.955882 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.955862 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:39.955999 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:39.955888 2575 scope.go:117] "RemoveContainer" containerID="6642f68feb4d3cae6d5f3fbeb6349f1a0cfa5b2784e06001c2ca1ddf777ac43f" Jul 12 17:03:40.871429 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:40.871397 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:03:40.963509 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:40.963475 2575 generic.go:356] "Generic (PLEG): container finished" podID="6a71645c-fb28-4ade-b3cf-d4d16e3259fa" containerID="52e1086c714b922fc671d6d4633a00e6de5a8dbe1bffdab37375c350b19d76fb" exitCode=0 Jul 12 17:03:40.963687 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:40.963565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309749413760" event={"ID":"6a71645c-fb28-4ade-b3cf-d4d16e3259fa","Type":"ContainerDied","Data":"52e1086c714b922fc671d6d4633a00e6de5a8dbe1bffdab37375c350b19d76fb"} Jul 12 17:03:42.117792 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.117763 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:42.137336 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.137304 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7387-140309749413760_6a71645c-fb28-4ade-b3cf-d4d16e3259fa/test-curl-7387-140309749413760/0.log" Jul 12 17:03:42.159519 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.159488 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:42.163511 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.163486 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7387-140309749413760"] Jul 12 17:03:42.175586 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.175563 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwlvl\" (UniqueName: \"kubernetes.io/projected/6a71645c-fb28-4ade-b3cf-d4d16e3259fa-kube-api-access-qwlvl\") pod \"6a71645c-fb28-4ade-b3cf-d4d16e3259fa\" (UID: \"6a71645c-fb28-4ade-b3cf-d4d16e3259fa\") " Jul 12 17:03:42.177595 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.177566 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a71645c-fb28-4ade-b3cf-d4d16e3259fa-kube-api-access-qwlvl" (OuterVolumeSpecName: "kube-api-access-qwlvl") pod "6a71645c-fb28-4ade-b3cf-d4d16e3259fa" (UID: "6a71645c-fb28-4ade-b3cf-d4d16e3259fa"). InnerVolumeSpecName "kube-api-access-qwlvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:03:42.277394 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.277313 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwlvl\" (UniqueName: \"kubernetes.io/projected/6a71645c-fb28-4ade-b3cf-d4d16e3259fa-kube-api-access-qwlvl\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:42.279588 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.279567 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:42.279961 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.279948 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a71645c-fb28-4ade-b3cf-d4d16e3259fa" containerName="test-curl-7387-140309749413760" Jul 12 17:03:42.280004 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.279962 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a71645c-fb28-4ade-b3cf-d4d16e3259fa" containerName="test-curl-7387-140309749413760" Jul 12 17:03:42.280051 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.280033 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a71645c-fb28-4ade-b3cf-d4d16e3259fa" containerName="test-curl-7387-140309749413760" Jul 12 17:03:42.283036 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.283010 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:42.286606 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.286579 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:42.378741 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.378706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n65r\" (UniqueName: \"kubernetes.io/projected/bedd003b-fd33-4073-8acc-eae3acb063b4-kube-api-access-9n65r\") pod \"test-curl-7387-140309749414768\" (UID: \"bedd003b-fd33-4073-8acc-eae3acb063b4\") " pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:42.479740 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.479710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n65r\" (UniqueName: \"kubernetes.io/projected/bedd003b-fd33-4073-8acc-eae3acb063b4-kube-api-access-9n65r\") pod \"test-curl-7387-140309749414768\" (UID: \"bedd003b-fd33-4073-8acc-eae3acb063b4\") " pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:42.487998 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.487964 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n65r\" (UniqueName: \"kubernetes.io/projected/bedd003b-fd33-4073-8acc-eae3acb063b4-kube-api-access-9n65r\") pod \"test-curl-7387-140309749414768\" (UID: \"bedd003b-fd33-4073-8acc-eae3acb063b4\") " pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:42.593823 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.593736 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:42.729147 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.729126 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:42.730532 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:03:42.730507 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbedd003b_fd33_4073_8acc_eae3acb063b4.slice/crio-8ec32780f7ddf0ee67cb38d2ab52ffcffa740f5344bf7aef12aee367bfccf81d WatchSource:0}: Error finding container 8ec32780f7ddf0ee67cb38d2ab52ffcffa740f5344bf7aef12aee367bfccf81d: Status 404 returned error can't find the container with id 8ec32780f7ddf0ee67cb38d2ab52ffcffa740f5344bf7aef12aee367bfccf81d Jul 12 17:03:42.977623 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.977587 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:42.977623 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.977622 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a857355d75166e2a5d172b1ad59a364c723b387d1c315452aab42eb0dd233344" Jul 12 17:03:42.978881 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:42.978854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309749414768" event={"ID":"bedd003b-fd33-4073-8acc-eae3acb063b4","Type":"ContainerStarted","Data":"8ec32780f7ddf0ee67cb38d2ab52ffcffa740f5344bf7aef12aee367bfccf81d"} Jul 12 17:03:42.988436 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:03:42.988407 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7387-140309749413760\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="6a71645c-fb28-4ade-b3cf-d4d16e3259fa" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:43.133370 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.133336 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" podUID="60e78df5-1cd4-4a24-afea-5ad3b566486d" containerName="main" containerID="cri-o://9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5" gracePeriod=45 Jul 12 17:03:43.394453 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.394423 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:03:43.396803 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:03:43.396776 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7387-140309749413760\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="6a71645c-fb28-4ade-b3cf-d4d16e3259fa" pod="openshift-ingress/test-curl-7387-140309749413760" Jul 12 17:03:43.488468 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488437 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-home\") pod \"60e78df5-1cd4-4a24-afea-5ad3b566486d\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " Jul 12 17:03:43.488653 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488496 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e78df5-1cd4-4a24-afea-5ad3b566486d-tls-certs\") pod \"60e78df5-1cd4-4a24-afea-5ad3b566486d\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " Jul 12 17:03:43.488653 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488520 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-model-cache\") pod \"60e78df5-1cd4-4a24-afea-5ad3b566486d\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " Jul 12 17:03:43.488653 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488567 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-tmp-dir\") pod \"60e78df5-1cd4-4a24-afea-5ad3b566486d\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " Jul 12 17:03:43.488653 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488594 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-dshm\") pod \"60e78df5-1cd4-4a24-afea-5ad3b566486d\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " Jul 12 17:03:43.488653 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488631 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5vjs\" (UniqueName: \"kubernetes.io/projected/60e78df5-1cd4-4a24-afea-5ad3b566486d-kube-api-access-j5vjs\") pod \"60e78df5-1cd4-4a24-afea-5ad3b566486d\" (UID: \"60e78df5-1cd4-4a24-afea-5ad3b566486d\") " Jul 12 17:03:43.488907 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488763 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-home" (OuterVolumeSpecName: "home") pod "60e78df5-1cd4-4a24-afea-5ad3b566486d" (UID: "60e78df5-1cd4-4a24-afea-5ad3b566486d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:03:43.488907 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488788 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-model-cache" (OuterVolumeSpecName: "model-cache") pod "60e78df5-1cd4-4a24-afea-5ad3b566486d" (UID: "60e78df5-1cd4-4a24-afea-5ad3b566486d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:03:43.488907 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488857 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60e78df5-1cd4-4a24-afea-5ad3b566486d" (UID: "60e78df5-1cd4-4a24-afea-5ad3b566486d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:03:43.489010 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488920 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-home\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:43.489010 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.488938 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-model-cache\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:43.490601 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.490581 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e78df5-1cd4-4a24-afea-5ad3b566486d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60e78df5-1cd4-4a24-afea-5ad3b566486d" (UID: "60e78df5-1cd4-4a24-afea-5ad3b566486d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:03:43.490715 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.490694 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-dshm" (OuterVolumeSpecName: "dshm") pod "60e78df5-1cd4-4a24-afea-5ad3b566486d" (UID: "60e78df5-1cd4-4a24-afea-5ad3b566486d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:03:43.490764 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.490741 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e78df5-1cd4-4a24-afea-5ad3b566486d-kube-api-access-j5vjs" (OuterVolumeSpecName: "kube-api-access-j5vjs") pod "60e78df5-1cd4-4a24-afea-5ad3b566486d" (UID: "60e78df5-1cd4-4a24-afea-5ad3b566486d"). InnerVolumeSpecName "kube-api-access-j5vjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:03:43.590312 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.590233 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-tmp-dir\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:43.590312 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.590265 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e78df5-1cd4-4a24-afea-5ad3b566486d-dshm\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:43.590312 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.590274 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5vjs\" (UniqueName: \"kubernetes.io/projected/60e78df5-1cd4-4a24-afea-5ad3b566486d-kube-api-access-j5vjs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:43.590312 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.590283 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e78df5-1cd4-4a24-afea-5ad3b566486d-tls-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:43.876372 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.876280 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a71645c-fb28-4ade-b3cf-d4d16e3259fa" path="/var/lib/kubelet/pods/6a71645c-fb28-4ade-b3cf-d4d16e3259fa/volumes" Jul 12 17:03:43.986264 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.986229 2575 generic.go:356] "Generic (PLEG): container finished" podID="60e78df5-1cd4-4a24-afea-5ad3b566486d" containerID="9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5" exitCode=0 Jul 12 17:03:43.986463 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.986280 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" Jul 12 17:03:43.986463 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.986320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" event={"ID":"60e78df5-1cd4-4a24-afea-5ad3b566486d","Type":"ContainerDied","Data":"9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5"} Jul 12 17:03:43.986463 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.986360 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n" event={"ID":"60e78df5-1cd4-4a24-afea-5ad3b566486d","Type":"ContainerDied","Data":"ec57bd4d1c6476543ec054ed9a1c153466a49c488f0b65a2dc0495e6d2fbf9f4"} Jul 12 17:03:43.986463 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.986385 2575 scope.go:117] "RemoveContainer" containerID="9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5" Jul 12 17:03:43.988060 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.988034 2575 generic.go:356] "Generic (PLEG): container finished" podID="bedd003b-fd33-4073-8acc-eae3acb063b4" containerID="6a12b43420572d448b55af022d22f5e60c73dda80f0e038f3b59c9b759796029" exitCode=0 Jul 12 17:03:43.988174 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.988082 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309749414768" event={"ID":"bedd003b-fd33-4073-8acc-eae3acb063b4","Type":"ContainerDied","Data":"6a12b43420572d448b55af022d22f5e60c73dda80f0e038f3b59c9b759796029"} Jul 12 17:03:43.995862 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.995828 2575 scope.go:117] "RemoveContainer" containerID="9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5" Jul 12 17:03:43.996101 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:03:43.996080 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5\": container with ID starting with 9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5 not found: ID does not exist" containerID="9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5" Jul 12 17:03:43.996145 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:43.996111 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5"} err="failed to get container status \"9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5\": rpc error: code = NotFound desc = could not find container \"9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5\": container with ID starting with 9f8e1fa38593bf09504c8ebcc26955a612a847ab343e1b7100cf2939873a8dc5 not found: ID does not exist" Jul 12 17:03:44.016460 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.016429 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n"] Jul 12 17:03:44.018715 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.018694 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-17d27e/rate-test-model-17d27e-kserve-655b97fd75-jvn2n"] Jul 12 17:03:44.077018 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.076975 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" podUID="81541e68-0fa3-4ce2-834d-8dd441c259ca" containerName="main" containerID="cri-o://584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19" gracePeriod=45 Jul 12 17:03:44.341005 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.340982 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:03:44.498719 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.498691 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-tmp-dir\") pod \"81541e68-0fa3-4ce2-834d-8dd441c259ca\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " Jul 12 17:03:44.498893 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.498753 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-dshm\") pod \"81541e68-0fa3-4ce2-834d-8dd441c259ca\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " Jul 12 17:03:44.498893 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.498798 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81541e68-0fa3-4ce2-834d-8dd441c259ca-tls-certs\") pod \"81541e68-0fa3-4ce2-834d-8dd441c259ca\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " Jul 12 17:03:44.498893 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.498834 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-home\") pod \"81541e68-0fa3-4ce2-834d-8dd441c259ca\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " Jul 12 17:03:44.498893 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.498878 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5zwf\" (UniqueName: \"kubernetes.io/projected/81541e68-0fa3-4ce2-834d-8dd441c259ca-kube-api-access-h5zwf\") pod \"81541e68-0fa3-4ce2-834d-8dd441c259ca\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " Jul 12 17:03:44.499100 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.498963 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-model-cache\") pod \"81541e68-0fa3-4ce2-834d-8dd441c259ca\" (UID: \"81541e68-0fa3-4ce2-834d-8dd441c259ca\") " Jul 12 17:03:44.499100 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.499040 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81541e68-0fa3-4ce2-834d-8dd441c259ca" (UID: "81541e68-0fa3-4ce2-834d-8dd441c259ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:03:44.499220 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.499098 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-home" (OuterVolumeSpecName: "home") pod "81541e68-0fa3-4ce2-834d-8dd441c259ca" (UID: "81541e68-0fa3-4ce2-834d-8dd441c259ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:03:44.499291 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.499237 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "81541e68-0fa3-4ce2-834d-8dd441c259ca" (UID: "81541e68-0fa3-4ce2-834d-8dd441c259ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:03:44.499342 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.499320 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-home\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:44.499378 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.499344 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-model-cache\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:44.499378 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.499364 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-tmp-dir\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:44.500902 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.500878 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81541e68-0fa3-4ce2-834d-8dd441c259ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81541e68-0fa3-4ce2-834d-8dd441c259ca" (UID: "81541e68-0fa3-4ce2-834d-8dd441c259ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 17:03:44.501013 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.500896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-dshm" (OuterVolumeSpecName: "dshm") pod "81541e68-0fa3-4ce2-834d-8dd441c259ca" (UID: "81541e68-0fa3-4ce2-834d-8dd441c259ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 17:03:44.501013 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.500963 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81541e68-0fa3-4ce2-834d-8dd441c259ca-kube-api-access-h5zwf" (OuterVolumeSpecName: "kube-api-access-h5zwf") pod "81541e68-0fa3-4ce2-834d-8dd441c259ca" (UID: "81541e68-0fa3-4ce2-834d-8dd441c259ca"). InnerVolumeSpecName "kube-api-access-h5zwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:03:44.600645 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.600593 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81541e68-0fa3-4ce2-834d-8dd441c259ca-dshm\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:44.600645 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.600638 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81541e68-0fa3-4ce2-834d-8dd441c259ca-tls-certs\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:44.600645 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.600650 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zwf\" (UniqueName: \"kubernetes.io/projected/81541e68-0fa3-4ce2-834d-8dd441c259ca-kube-api-access-h5zwf\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:44.996178 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.996145 2575 generic.go:356] "Generic (PLEG): container finished" podID="81541e68-0fa3-4ce2-834d-8dd441c259ca" containerID="584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19" exitCode=0 Jul 12 17:03:44.996178 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.996185 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" Jul 12 17:03:44.996427 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.996225 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" event={"ID":"81541e68-0fa3-4ce2-834d-8dd441c259ca","Type":"ContainerDied","Data":"584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19"} Jul 12 17:03:44.996427 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.996274 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v" event={"ID":"81541e68-0fa3-4ce2-834d-8dd441c259ca","Type":"ContainerDied","Data":"a18be02cc3cac5dd6d2f0e737c0babcb9320dcf207972875fe45746dd96b955c"} Jul 12 17:03:44.996427 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:44.996299 2575 scope.go:117] "RemoveContainer" containerID="584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19" Jul 12 17:03:45.006772 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.006751 2575 scope.go:117] "RemoveContainer" containerID="584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19" Jul 12 17:03:45.007104 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:03:45.007086 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19\": container with ID starting with 584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19 not found: ID does not exist" containerID="584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19" Jul 12 17:03:45.007169 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.007115 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19"} err="failed to get container status \"584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19\": rpc error: code = NotFound desc = could not find container \"584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19\": container with ID starting with 584519d42f8110a6c8f8aa7b70a045f05a24d886cf9a4f5729358579b027ce19 not found: ID does not exist" Jul 12 17:03:45.017360 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.017327 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v"] Jul 12 17:03:45.020551 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.020525 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cc457c/rate-test-model-cc457c-kserve-64d777958c-z5w9v"] Jul 12 17:03:45.148164 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.148137 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:45.165860 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.165815 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7387-140309749414768_bedd003b-fd33-4073-8acc-eae3acb063b4/test-curl-7387-140309749414768/0.log" Jul 12 17:03:45.188176 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.188149 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:45.190442 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.190421 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:45.307632 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.307541 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n65r\" (UniqueName: \"kubernetes.io/projected/bedd003b-fd33-4073-8acc-eae3acb063b4-kube-api-access-9n65r\") pod \"bedd003b-fd33-4073-8acc-eae3acb063b4\" (UID: \"bedd003b-fd33-4073-8acc-eae3acb063b4\") " Jul 12 17:03:45.310205 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310158 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedd003b-fd33-4073-8acc-eae3acb063b4-kube-api-access-9n65r" (OuterVolumeSpecName: "kube-api-access-9n65r") pod "bedd003b-fd33-4073-8acc-eae3acb063b4" (UID: "bedd003b-fd33-4073-8acc-eae3acb063b4"). InnerVolumeSpecName "kube-api-access-9n65r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:03:45.310357 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310229 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:45.310641 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310626 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e78df5-1cd4-4a24-afea-5ad3b566486d" containerName="main" Jul 12 17:03:45.310708 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310641 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e78df5-1cd4-4a24-afea-5ad3b566486d" containerName="main" Jul 12 17:03:45.310708 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310667 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bedd003b-fd33-4073-8acc-eae3acb063b4" containerName="test-curl-7387-140309749414768" Jul 12 17:03:45.310708 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310673 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bedd003b-fd33-4073-8acc-eae3acb063b4" containerName="test-curl-7387-140309749414768" Jul 12 17:03:45.310708 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310683 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81541e68-0fa3-4ce2-834d-8dd441c259ca" containerName="main" Jul 12 17:03:45.310708 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310688 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="81541e68-0fa3-4ce2-834d-8dd441c259ca" containerName="main" Jul 12 17:03:45.310952 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310747 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="81541e68-0fa3-4ce2-834d-8dd441c259ca" containerName="main" Jul 12 17:03:45.310952 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310755 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bedd003b-fd33-4073-8acc-eae3acb063b4" containerName="test-curl-7387-140309749414768" Jul 12 17:03:45.310952 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.310762 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e78df5-1cd4-4a24-afea-5ad3b566486d" containerName="main" Jul 12 17:03:45.313691 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.313672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:45.320035 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.320012 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:45.408546 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.408510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk5qd\" (UniqueName: \"kubernetes.io/projected/14ab7b29-f63a-4ca4-a547-a50488434a9d-kube-api-access-mk5qd\") pod \"test-curl-7387-140309749414768\" (UID: \"14ab7b29-f63a-4ca4-a547-a50488434a9d\") " pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:45.408921 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.408575 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n65r\" (UniqueName: \"kubernetes.io/projected/bedd003b-fd33-4073-8acc-eae3acb063b4-kube-api-access-9n65r\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:45.509369 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.509332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk5qd\" (UniqueName: \"kubernetes.io/projected/14ab7b29-f63a-4ca4-a547-a50488434a9d-kube-api-access-mk5qd\") pod \"test-curl-7387-140309749414768\" (UID: \"14ab7b29-f63a-4ca4-a547-a50488434a9d\") " pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:45.517343 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.517317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk5qd\" (UniqueName: \"kubernetes.io/projected/14ab7b29-f63a-4ca4-a547-a50488434a9d-kube-api-access-mk5qd\") pod \"test-curl-7387-140309749414768\" (UID: \"14ab7b29-f63a-4ca4-a547-a50488434a9d\") " pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:45.629342 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.629262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:45.763800 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.763776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:45.765543 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:03:45.765514 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14ab7b29_f63a_4ca4_a547_a50488434a9d.slice/crio-546ea10857280a5ccad7a4adc87b6cbc6dce7b944c82dca8f8660443de6b4dea WatchSource:0}: Error finding container 546ea10857280a5ccad7a4adc87b6cbc6dce7b944c82dca8f8660443de6b4dea: Status 404 returned error can't find the container with id 546ea10857280a5ccad7a4adc87b6cbc6dce7b944c82dca8f8660443de6b4dea Jul 12 17:03:45.879584 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.879500 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e78df5-1cd4-4a24-afea-5ad3b566486d" path="/var/lib/kubelet/pods/60e78df5-1cd4-4a24-afea-5ad3b566486d/volumes" Jul 12 17:03:45.879913 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.879899 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81541e68-0fa3-4ce2-834d-8dd441c259ca" path="/var/lib/kubelet/pods/81541e68-0fa3-4ce2-834d-8dd441c259ca/volumes" Jul 12 17:03:45.880224 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:45.880184 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bedd003b-fd33-4073-8acc-eae3acb063b4" path="/var/lib/kubelet/pods/bedd003b-fd33-4073-8acc-eae3acb063b4/volumes" Jul 12 17:03:46.003862 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:46.003828 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:46.004020 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:46.003887 2575 scope.go:117] "RemoveContainer" containerID="6a12b43420572d448b55af022d22f5e60c73dda80f0e038f3b59c9b759796029" Jul 12 17:03:46.005108 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:46.005083 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309749414768" event={"ID":"14ab7b29-f63a-4ca4-a547-a50488434a9d","Type":"ContainerStarted","Data":"546ea10857280a5ccad7a4adc87b6cbc6dce7b944c82dca8f8660443de6b4dea"} Jul 12 17:03:47.012740 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:47.012707 2575 generic.go:356] "Generic (PLEG): container finished" podID="14ab7b29-f63a-4ca4-a547-a50488434a9d" containerID="3c1b694a1dbfcdc47fb2911eb8d42b71c167a8163959685915956912de1c9445" exitCode=0 Jul 12 17:03:47.013140 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:47.012757 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7387-140309749414768" event={"ID":"14ab7b29-f63a-4ca4-a547-a50488434a9d","Type":"ContainerDied","Data":"3c1b694a1dbfcdc47fb2911eb8d42b71c167a8163959685915956912de1c9445"} Jul 12 17:03:48.165137 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:48.165115 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:48.183718 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:48.183692 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7387-140309749414768_14ab7b29-f63a-4ca4-a547-a50488434a9d/test-curl-7387-140309749414768/0.log" Jul 12 17:03:48.205837 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:48.205801 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:48.210575 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:48.210546 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7387-140309749414768"] Jul 12 17:03:48.333713 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:48.333631 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk5qd\" (UniqueName: \"kubernetes.io/projected/14ab7b29-f63a-4ca4-a547-a50488434a9d-kube-api-access-mk5qd\") pod \"14ab7b29-f63a-4ca4-a547-a50488434a9d\" (UID: \"14ab7b29-f63a-4ca4-a547-a50488434a9d\") " Jul 12 17:03:48.335701 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:48.335665 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ab7b29-f63a-4ca4-a547-a50488434a9d-kube-api-access-mk5qd" (OuterVolumeSpecName: "kube-api-access-mk5qd") pod "14ab7b29-f63a-4ca4-a547-a50488434a9d" (UID: "14ab7b29-f63a-4ca4-a547-a50488434a9d"). InnerVolumeSpecName "kube-api-access-mk5qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 17:03:48.434562 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:48.434524 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk5qd\" (UniqueName: \"kubernetes.io/projected/14ab7b29-f63a-4ca4-a547-a50488434a9d-kube-api-access-mk5qd\") on node \"ip-10-0-134-123.ec2.internal\" DevicePath \"\"" Jul 12 17:03:48.533647 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:48.533606 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:49.007103 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:49.007064 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:49.031802 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:49.031772 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:49.031942 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:49.031772 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="546ea10857280a5ccad7a4adc87b6cbc6dce7b944c82dca8f8660443de6b4dea" Jul 12 17:03:49.041895 ip-10-0-134-123 kubenswrapper[2575]: E0712 17:03:49.041859 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7387-140309749414768\" is forbidden: User \"system:node:ip-10-0-134-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-123.ec2.internal' and this object" podUID="14ab7b29-f63a-4ca4-a547-a50488434a9d" pod="openshift-ingress/test-curl-7387-140309749414768" Jul 12 17:03:49.105469 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:49.105437 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:49.206096 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:49.206066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:49.357821 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:49.357729 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:49.875320 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:49.875284 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ab7b29-f63a-4ca4-a547-a50488434a9d" path="/var/lib/kubelet/pods/14ab7b29-f63a-4ca4-a547-a50488434a9d/volumes" Jul 12 17:03:54.851568 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:54.851475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:54.952001 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:54.951969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:55.110301 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:55.110171 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:55.344812 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:55.344767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:55.470184 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:55.470145 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:03:55.500771 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:55.500735 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:55.906814 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:55.906714 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:56.007056 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:56.007013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:56.106834 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:56.106796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:03:56.216632 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:03:56.216600 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:01.643903 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:01.643873 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:01.704751 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:01.704709 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:01.923051 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:01.922965 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:02.136275 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:02.136228 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:04.761725 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:04.761697 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jtvlq"] Jul 12 17:04:07.381142 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:07.381099 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9s9vl_e1c14d57-3002-459b-864c-c72ea58b2bc3/manager/0.log" Jul 12 17:04:07.505489 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:07.505459 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2ngrd_5f3aeff8-d6d6-43a5-82bd-505568ed4002/manager/0.log" Jul 12 17:04:07.742794 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:07.742764 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x926v_e9ed3e70-852f-4188-8568-0e724d84eef6/server/0.log" Jul 12 17:04:07.866357 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:07.866313 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wccbh_00b243a7-1af7-4dbf-9ca3-5a69756185fa/manager/0.log" Jul 12 17:04:07.972138 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:07.972113 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869b4c4546-2x5pv_142a09a0-7896-4aed-bfe0-6001e4ad09d3/copy-manifests/0.log" Jul 12 17:04:07.978822 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:07.978791 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869b4c4546-2x5pv_142a09a0-7896-4aed-bfe0-6001e4ad09d3/manager/0.log" Jul 12 17:04:09.469376 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.469348 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6_072b95d8-8881-4843-9233-788aeea3645e/extract/0.log" Jul 12 17:04:09.475320 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.475289 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6_072b95d8-8881-4843-9233-788aeea3645e/util/0.log" Jul 12 17:04:09.483560 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.483538 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6_072b95d8-8881-4843-9233-788aeea3645e/pull/0.log" Jul 12 17:04:09.593857 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.593769 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8_373f3c02-0b0a-44dc-910f-c05996b565ac/util/0.log" Jul 12 17:04:09.599947 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.599924 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8_373f3c02-0b0a-44dc-910f-c05996b565ac/pull/0.log" Jul 12 17:04:09.606248 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.606222 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8_373f3c02-0b0a-44dc-910f-c05996b565ac/extract/0.log" Jul 12 17:04:09.724646 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.724618 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr_43b000cb-a0da-49ed-b7e5-2400ceeb46cb/pull/0.log" Jul 12 17:04:09.731639 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.731617 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr_43b000cb-a0da-49ed-b7e5-2400ceeb46cb/extract/0.log" Jul 12 17:04:09.737869 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.737848 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr_43b000cb-a0da-49ed-b7e5-2400ceeb46cb/util/0.log" Jul 12 17:04:09.843309 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.843281 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg_ad922c3a-ed87-48e7-b3c7-1014cb8a1866/util/0.log" Jul 12 17:04:09.849109 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.849048 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg_ad922c3a-ed87-48e7-b3c7-1014cb8a1866/pull/0.log" Jul 12 17:04:09.854546 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:09.854522 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg_ad922c3a-ed87-48e7-b3c7-1014cb8a1866/extract/0.log" Jul 12 17:04:10.209409 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:10.209382 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pmbpx_b4ab23b1-52f4-46d2-bbdf-256e4929b8a0/manager/0.log" Jul 12 17:04:10.318286 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:10.318260 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-k44fs_3a386d73-e480-4fee-a740-19265d43ccb5/kuadrant-console-plugin/0.log" Jul 12 17:04:10.439513 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:10.439483 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dkpsc_49de7d40-8e73-4a34-b971-a3a95a42faab/registry-server/0.log" Jul 12 17:04:10.561176 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:10.561087 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vmkh8_5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd/manager/0.log" Jul 12 17:04:10.672635 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:10.672606 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jtvlq_88826bf9-5f86-4544-bed2-ea8f9aa135e9/limitador/0.log" Jul 12 17:04:10.800388 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:10.800356 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-7wth6_afe1243b-90d8-494c-a826-8f974945c7c2/manager/0.log" Jul 12 17:04:11.150737 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:11.150702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x7nw8_cd8057ab-8e02-4acf-8527-e8af17338ede/istio-proxy/0.log" Jul 12 17:04:11.494870 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:11.494834 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545795f48c-vwbng_923f08c1-c7f8-4f05-b561-f38e4cfaa1e7/kube-auth-proxy/0.log" Jul 12 17:04:11.618927 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:11.618897 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jlcst_5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956/istio-proxy/1.log" Jul 12 17:04:12.438336 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:12.438308 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-klpgx_dfe93d17-d8e0-4441-897b-d56f51a8e346/main/0.log" Jul 12 17:04:12.665626 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:12.665595 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzx526_56464e90-ccb8-43bf-b58d-e2cf7b96cb6a/main/0.log" Jul 12 17:04:12.888168 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:12.888088 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-gpnnk_4291ea18-6d1a-4ab2-b67c-24f70f6b0044/main/0.log" Jul 12 17:04:16.830347 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:16.830316 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:16.930204 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:16.930146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" limit=1024 Jul 12 17:04:19.538441 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:19.538406 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4z46t_97442fa3-b0da-4f2d-a0a7-b5682b9c2402/global-pull-secret-syncer/0.log" Jul 12 17:04:19.673248 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:19.673216 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5r44g_99b9a17e-370f-43bc-8186-fe34fe8a47c6/konnectivity-agent/0.log" Jul 12 17:04:19.789020 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:19.788924 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-123.ec2.internal_4b6203c45afbfc7cc0bd349ce0d548f2/haproxy/0.log" Jul 12 17:04:24.366254 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.366224 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6_072b95d8-8881-4843-9233-788aeea3645e/extract/0.log" Jul 12 17:04:24.391739 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.391713 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6_072b95d8-8881-4843-9233-788aeea3645e/util/0.log" Jul 12 17:04:24.411801 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.411770 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596s7f6_072b95d8-8881-4843-9233-788aeea3645e/pull/0.log" Jul 12 17:04:24.438866 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.438844 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8_373f3c02-0b0a-44dc-910f-c05996b565ac/extract/0.log" Jul 12 17:04:24.457812 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.457783 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8_373f3c02-0b0a-44dc-910f-c05996b565ac/util/0.log" Jul 12 17:04:24.483106 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.483081 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0tqvg8_373f3c02-0b0a-44dc-910f-c05996b565ac/pull/0.log" Jul 12 17:04:24.512950 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.512914 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr_43b000cb-a0da-49ed-b7e5-2400ceeb46cb/extract/0.log" Jul 12 17:04:24.532994 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.532961 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr_43b000cb-a0da-49ed-b7e5-2400ceeb46cb/util/0.log" Jul 12 17:04:24.554290 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.554267 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7374sfr_43b000cb-a0da-49ed-b7e5-2400ceeb46cb/pull/0.log" Jul 12 17:04:24.579427 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.579407 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg_ad922c3a-ed87-48e7-b3c7-1014cb8a1866/extract/0.log" Jul 12 17:04:24.602549 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.602526 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg_ad922c3a-ed87-48e7-b3c7-1014cb8a1866/util/0.log" Jul 12 17:04:24.625962 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:24.625887 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kk8sg_ad922c3a-ed87-48e7-b3c7-1014cb8a1866/pull/0.log" Jul 12 17:04:25.197400 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:25.197373 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pmbpx_b4ab23b1-52f4-46d2-bbdf-256e4929b8a0/manager/0.log" Jul 12 17:04:25.220346 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:25.220317 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-k44fs_3a386d73-e480-4fee-a740-19265d43ccb5/kuadrant-console-plugin/0.log" Jul 12 17:04:25.283958 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:25.283925 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dkpsc_49de7d40-8e73-4a34-b971-a3a95a42faab/registry-server/0.log" Jul 12 17:04:25.446994 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:25.446964 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vmkh8_5d2fcddd-cb72-47b9-a9ce-5cc660cb8dbd/manager/0.log" Jul 12 17:04:25.468516 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:25.468449 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jtvlq_88826bf9-5f86-4544-bed2-ea8f9aa135e9/limitador/0.log" Jul 12 17:04:25.570333 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:25.570303 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-7wth6_afe1243b-90d8-494c-a826-8f974945c7c2/manager/0.log" Jul 12 17:04:27.643999 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:27.643971 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7ccb4cc598-7crw7_cfb12230-4caa-4739-88b6-16cdb37985fb/metrics-server/0.log" Jul 12 17:04:27.672620 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:27.672594 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-bj27x_0a6b5ac1-015a-4aa3-955e-fb2341b81ead/monitoring-plugin/0.log" Jul 12 17:04:27.847713 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:27.847686 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-glkh9_98c3b438-740e-48e7-950e-ce0ad2ada902/node-exporter/0.log" Jul 12 17:04:27.865323 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:27.865299 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-glkh9_98c3b438-740e-48e7-950e-ce0ad2ada902/kube-rbac-proxy/0.log" Jul 12 17:04:27.883609 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:27.883589 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-glkh9_98c3b438-740e-48e7-950e-ce0ad2ada902/init-textfile/0.log" Jul 12 17:04:27.908251 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:27.908179 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-8d645_b281d074-ea80-499f-9b85-27da239f3c69/kube-rbac-proxy-main/0.log" Jul 12 17:04:27.928481 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:27.928458 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-8d645_b281d074-ea80-499f-9b85-27da239f3c69/kube-rbac-proxy-self/0.log" Jul 12 17:04:27.953016 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:27.952995 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-8d645_b281d074-ea80-499f-9b85-27da239f3c69/openshift-state-metrics/0.log" Jul 12 17:04:28.146682 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:28.146653 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-8ld92_b79d1dd6-8d16-4388-9ccc-2c896ac5487f/prometheus-operator/0.log" Jul 12 17:04:28.171314 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:28.171245 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-8ld92_b79d1dd6-8d16-4388-9ccc-2c896ac5487f/kube-rbac-proxy/0.log" Jul 12 17:04:28.194293 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:28.194268 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-g5s49_c7fccd46-c827-4d70-827d-70d54b527970/prometheus-operator-admission-webhook/0.log" Jul 12 17:04:28.219254 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:28.219231 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-644bcfc9c8-c4qxq_eb71236f-3c31-4c39-b118-481c18e8e516/telemeter-client/0.log" Jul 12 17:04:28.244596 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:28.244578 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-644bcfc9c8-c4qxq_eb71236f-3c31-4c39-b118-481c18e8e516/reload/0.log" Jul 12 17:04:28.264532 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:28.264509 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-644bcfc9c8-c4qxq_eb71236f-3c31-4c39-b118-481c18e8e516/kube-rbac-proxy/0.log" Jul 12 17:04:29.505322 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.505290 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg"] Jul 12 17:04:29.505686 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.505674 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ab7b29-f63a-4ca4-a547-a50488434a9d" containerName="test-curl-7387-140309749414768" Jul 12 17:04:29.505727 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.505707 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ab7b29-f63a-4ca4-a547-a50488434a9d" containerName="test-curl-7387-140309749414768" Jul 12 17:04:29.505781 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.505770 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ab7b29-f63a-4ca4-a547-a50488434a9d" containerName="test-curl-7387-140309749414768" Jul 12 17:04:29.509900 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.509883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.512558 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.512518 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-74brq\"/\"kube-root-ca.crt\"" Jul 12 17:04:29.512693 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.512558 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-74brq\"/\"default-dockercfg-w66hb\"" Jul 12 17:04:29.512693 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.512524 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-74brq\"/\"openshift-service-ca.crt\"" Jul 12 17:04:29.518481 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.518457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg"] Jul 12 17:04:29.614925 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.614890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-lib-modules\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.615113 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.614942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-podres\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.615113 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.615034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgv66\" (UniqueName: \"kubernetes.io/projected/e0b7226a-e881-499d-a551-d81a7d35accb-kube-api-access-fgv66\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.615113 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.615096 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-proc\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.615242 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.615128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-sys\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.715779 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.715745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-podres\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.715958 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.715791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv66\" (UniqueName: \"kubernetes.io/projected/e0b7226a-e881-499d-a551-d81a7d35accb-kube-api-access-fgv66\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.715958 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.715836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-proc\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.715958 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.715855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-sys\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.715958 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.715890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-lib-modules\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.715958 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.715915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-podres\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.716184 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.715965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-sys\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.716184 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.715965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-proc\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.716184 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.716045 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0b7226a-e881-499d-a551-d81a7d35accb-lib-modules\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.724554 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.724523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgv66\" (UniqueName: \"kubernetes.io/projected/e0b7226a-e881-499d-a551-d81a7d35accb-kube-api-access-fgv66\") pod \"perf-node-gather-daemonset-9jvlg\" (UID: \"e0b7226a-e881-499d-a551-d81a7d35accb\") " pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.821500 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.821404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:29.957993 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:29.957967 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg"] Jul 12 17:04:29.959710 ip-10-0-134-123 kubenswrapper[2575]: W0712 17:04:29.959668 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode0b7226a_e881_499d_a551_d81a7d35accb.slice/crio-4dbbc04da5f300f65263742d69a8a1d7a18f1fe24f3441016ac50aea811f1ab3 WatchSource:0}: Error finding container 4dbbc04da5f300f65263742d69a8a1d7a18f1fe24f3441016ac50aea811f1ab3: Status 404 returned error can't find the container with id 4dbbc04da5f300f65263742d69a8a1d7a18f1fe24f3441016ac50aea811f1ab3 Jul 12 17:04:30.263551 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:30.263518 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" event={"ID":"e0b7226a-e881-499d-a551-d81a7d35accb","Type":"ContainerStarted","Data":"b5d1ec2d08ee1935d129df813a17d7d494e975c2c69b2c6297a4b0c7a57829f3"} Jul 12 17:04:30.263714 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:30.263574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" event={"ID":"e0b7226a-e881-499d-a551-d81a7d35accb","Type":"ContainerStarted","Data":"4dbbc04da5f300f65263742d69a8a1d7a18f1fe24f3441016ac50aea811f1ab3"} Jul 12 17:04:30.263792 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:30.263714 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:30.279558 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:30.279506 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" podStartSLOduration=1.279488922 podStartE2EDuration="1.279488922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 17:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 17:04:30.279209398 +0000 UTC m=+2369.048637294" watchObservedRunningTime="2026-07-12 17:04:30.279488922 +0000 UTC m=+2369.048916820" Jul 12 17:04:30.771362 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:30.771334 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-f6b6877c8-4jgqg_d86f6fa7-2239-4cbc-93f6-db788a412edf/console/0.log" Jul 12 17:04:32.370758 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:32.370726 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6snlj_6b0dfc77-915e-424b-959b-ab873bf51dbe/dns/0.log" Jul 12 17:04:32.391262 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:32.391240 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6snlj_6b0dfc77-915e-424b-959b-ab873bf51dbe/kube-rbac-proxy/0.log" Jul 12 17:04:32.531100 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:32.531074 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6vlfb_cdd8c0d4-657b-41e0-a1f7-ed1c9e35d9ed/dns-node-resolver/0.log" Jul 12 17:04:33.121118 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:33.121097 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c52dt_dd0c6f87-ae63-46bc-9373-da4088aeba89/node-ca/0.log" Jul 12 17:04:34.090557 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:34.090521 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x7nw8_cd8057ab-8e02-4acf-8527-e8af17338ede/istio-proxy/0.log" Jul 12 17:04:34.279791 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:34.279762 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545795f48c-vwbng_923f08c1-c7f8-4f05-b561-f38e4cfaa1e7/kube-auth-proxy/0.log" Jul 12 17:04:34.309020 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:34.308990 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jlcst_5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956/istio-proxy/0.log" Jul 12 17:04:34.364247 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:34.364151 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jlcst_5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956/istio-proxy/1.log" Jul 12 17:04:35.162624 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:35.162597 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8hnw8_0e93a099-983c-4ae5-8d65-b25c4d30884b/serve-healthcheck-canary/0.log" Jul 12 17:04:35.841429 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:35.841401 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-th26f_774f43ee-8efb-4b23-9ffc-5d6ee08f1d18/kube-rbac-proxy/0.log" Jul 12 17:04:35.863135 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:35.863113 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-th26f_774f43ee-8efb-4b23-9ffc-5d6ee08f1d18/exporter/0.log" Jul 12 17:04:35.884575 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:35.884551 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-th26f_774f43ee-8efb-4b23-9ffc-5d6ee08f1d18/extractor/0.log" Jul 12 17:04:36.278604 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:36.278580 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-74brq/perf-node-gather-daemonset-9jvlg" Jul 12 17:04:38.054092 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:38.054055 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9s9vl_e1c14d57-3002-459b-864c-c72ea58b2bc3/manager/0.log" Jul 12 17:04:38.295429 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:38.295395 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2ngrd_5f3aeff8-d6d6-43a5-82bd-505568ed4002/manager/0.log" Jul 12 17:04:38.471379 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:38.471353 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x926v_e9ed3e70-852f-4188-8568-0e724d84eef6/server/0.log" Jul 12 17:04:38.575204 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:38.575133 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wccbh_00b243a7-1af7-4dbf-9ca3-5a69756185fa/manager/0.log" Jul 12 17:04:38.602025 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:38.601995 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869b4c4546-2x5pv_142a09a0-7896-4aed-bfe0-6001e4ad09d3/manager/0.log" Jul 12 17:04:38.624158 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:38.624135 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869b4c4546-2x5pv_142a09a0-7896-4aed-bfe0-6001e4ad09d3/copy-manifests/0.log" Jul 12 17:04:40.125997 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:40.125959 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-c49f2_5d3d199d-4eef-4182-b4c0-0c256ff3c324/openshift-lws-operator/0.log" Jul 12 17:04:40.316324 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:40.316299 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x7nw8_cd8057ab-8e02-4acf-8527-e8af17338ede/istio-proxy/0.log" Jul 12 17:04:40.484533 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:40.484501 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545795f48c-vwbng_923f08c1-c7f8-4f05-b561-f38e4cfaa1e7/kube-auth-proxy/0.log" Jul 12 17:04:40.505239 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:40.505175 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jlcst_5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956/istio-proxy/0.log" Jul 12 17:04:40.564398 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:40.564368 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jlcst_5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956/istio-proxy/1.log" Jul 12 17:04:41.787793 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:41.787767 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x7nw8_cd8057ab-8e02-4acf-8527-e8af17338ede/istio-proxy/0.log" Jul 12 17:04:41.942789 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:41.942761 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545795f48c-vwbng_923f08c1-c7f8-4f05-b561-f38e4cfaa1e7/kube-auth-proxy/0.log" Jul 12 17:04:41.965077 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:41.965048 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jlcst_5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956/istio-proxy/0.log" Jul 12 17:04:42.028968 ip-10-0-134-123 kubenswrapper[2575]: I0712 17:04:42.028939 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jlcst_5a4af49d-2c7c-4dbc-9e7a-4bd677a0a956/istio-proxy/1.log"