Jul 31 00:06:30.347447 ip-10-0-134-191 systemd[1]: Starting Kubernetes Kubelet... Jul 31 00:06:30.752423 ip-10-0-134-191 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 00:06:30.752423 ip-10-0-134-191 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 00:06:30.752423 ip-10-0-134-191 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 00:06:30.752423 ip-10-0-134-191 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 00:06:30.752423 ip-10-0-134-191 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 00:06:30.753580 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.753450 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 00:06:30.759322 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759290 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 00:06:30.759322 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759315 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 00:06:30.759322 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759319 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 00:06:30.759322 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759325 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 00:06:30.759322 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759329 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 00:06:30.759322 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759333 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 00:06:30.759322 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759336 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759339 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759343 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759346 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759349 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759352 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759355 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759359 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759363 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759366 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759369 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759372 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759375 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759378 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759382 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759385 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759390 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759395 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759398 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 00:06:30.759612 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759401 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759404 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759407 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759410 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759413 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759424 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759427 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759431 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759434 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759437 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759440 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759443 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759446 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759449 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759451 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759455 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759458 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759461 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759464 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759466 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 00:06:30.760105 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759469 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759472 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759475 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759478 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759481 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759484 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759487 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759490 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759494 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759496 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759499 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759502 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759505 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759507 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759510 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759513 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759516 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759518 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759521 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 00:06:30.760674 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759525 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759528 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759531 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759534 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759537 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759540 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759543 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759546 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759551 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759553 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759556 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759559 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759562 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759565 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759567 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759570 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759573 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759576 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759578 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759589 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 00:06:30.761200 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759592 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759595 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759597 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759600 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759603 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759606 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759609 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759612 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759614 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759617 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759621 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759624 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759627 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759630 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.759633 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760456 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760464 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760467 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760473 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 00:06:30.761736 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760477 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760481 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760485 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760488 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760491 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760495 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760498 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760501 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760504 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760507 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760511 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760513 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760516 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760520 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760523 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760526 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760529 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760532 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760536 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760540 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 00:06:30.762266 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760544 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760547 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760549 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760552 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760556 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760559 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760562 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760565 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760568 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760571 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760574 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760577 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760580 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760583 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760587 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760590 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760592 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760595 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760598 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760601 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 00:06:30.762806 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760604 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760606 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760609 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760612 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760615 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760618 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760622 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760624 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760627 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760630 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760632 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760635 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760640 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760642 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760645 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760648 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760652 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760655 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760658 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760660 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 00:06:30.763392 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760663 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760666 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760669 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760671 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760674 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760677 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760682 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760685 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760688 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760691 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760694 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760697 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760700 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760702 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760707 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760710 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760714 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760717 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760721 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 00:06:30.763930 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760725 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760731 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760734 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760738 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760741 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760743 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760746 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760749 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760752 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760755 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760758 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760761 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760763 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760766 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760770 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.760772 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761243 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761250 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761255 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 00:06:30.764463 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761259 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761262 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761266 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761269 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761272 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761275 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761278 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761281 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761285 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761288 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761291 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761295 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761298 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761301 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761306 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761309 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761312 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761315 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761318 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761321 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761324 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 00:06:30.764973 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761327 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761330 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761333 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761336 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761341 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761344 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761347 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761349 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761352 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761355 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761358 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761361 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761364 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761366 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761369 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761372 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761375 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761378 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761381 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761384 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 00:06:30.765557 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761387 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761390 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761393 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761395 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761398 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761403 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761406 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761409 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761412 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761415 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761417 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761442 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761446 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761449 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761453 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761457 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761462 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761465 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761468 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 00:06:30.766077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761470 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761473 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761476 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761479 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761482 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761484 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761487 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761490 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761493 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761496 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761498 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761501 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761504 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761508 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761514 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761517 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761519 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761522 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761527 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 00:06:30.766608 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761534 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761536 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761539 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761542 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761545 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761548 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761551 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761553 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761556 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761559 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761563 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761566 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761569 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761572 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761575 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761577 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.761580 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761666 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761678 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761689 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761696 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 00:06:30.767108 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761702 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761707 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761714 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761719 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761723 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761726 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761730 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761733 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761737 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761740 2575 flags.go:64] FLAG: --cgroup-root="" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761746 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761750 2575 flags.go:64] FLAG: --client-ca-file="" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761754 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761757 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761762 2575 flags.go:64] FLAG: --cluster-domain="" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761766 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761770 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761773 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761777 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761781 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761784 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761789 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761792 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761796 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 31 00:06:30.767725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761799 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761802 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761806 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761809 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761822 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761826 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761829 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761832 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761835 2575 flags.go:64] FLAG: --enable-server="true" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761839 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761844 2575 flags.go:64] FLAG: --event-burst="100" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761847 2575 flags.go:64] FLAG: --event-qps="50" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761850 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761854 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761857 2575 flags.go:64] FLAG: --eviction-hard="" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761861 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761864 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761867 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761871 2575 flags.go:64] FLAG: --eviction-soft="" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761874 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761877 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761881 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761885 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761888 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761892 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 00:06:30.768377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761895 2575 flags.go:64] FLAG: --feature-gates="" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761899 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761902 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761905 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761909 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761912 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761916 2575 flags.go:64] FLAG: --help="false" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761919 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-191.ec2.internal" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761923 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761926 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761930 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761933 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761937 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761941 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761944 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761947 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761950 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761953 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761956 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761959 2575 flags.go:64] FLAG: --kube-reserved="" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761962 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761966 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761969 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761972 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 00:06:30.769092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761975 2575 flags.go:64] FLAG: --lock-file="" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761978 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761982 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761985 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761991 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761996 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.761999 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762002 2575 flags.go:64] FLAG: --logging-format="text" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762006 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762009 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762012 2575 flags.go:64] FLAG: --manifest-url="" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762016 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762020 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762023 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762028 2575 flags.go:64] FLAG: --max-pods="110" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762031 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762034 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762037 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762041 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762044 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762047 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762050 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762059 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762062 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762066 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 00:06:30.769785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762069 2575 flags.go:64] FLAG: --pod-cidr="" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762072 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762078 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762081 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762085 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762088 2575 flags.go:64] FLAG: --port="10250" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762092 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762095 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0be399628578943ce" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762098 2575 flags.go:64] FLAG: --qos-reserved="" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762101 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762105 2575 flags.go:64] FLAG: --register-node="true" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762108 2575 flags.go:64] FLAG: --register-with-taints="" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762112 2575 flags.go:64] FLAG: --registry-burst="10" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762116 2575 flags.go:64] FLAG: --registry-qps="5" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762119 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762122 2575 flags.go:64] FLAG: --reserved-memory="" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762126 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762130 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762133 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762136 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762139 2575 flags.go:64] FLAG: --runonce="false" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762142 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762146 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762149 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762152 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 00:06:30.770493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762155 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762158 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762161 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762164 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762167 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762171 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762174 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762177 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762180 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762197 2575 flags.go:64] FLAG: --system-cgroups="" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762201 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762207 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762210 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762213 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762218 2575 flags.go:64] FLAG: --tls-min-version="" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762221 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762224 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762228 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762231 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762234 2575 flags.go:64] FLAG: --v="2" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762239 2575 flags.go:64] FLAG: --version="false" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762244 2575 flags.go:64] FLAG: --vmodule="" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762248 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.762251 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762384 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 00:06:30.771165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762389 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762395 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762399 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762402 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762405 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762408 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762410 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762413 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762417 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762420 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762423 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762425 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762430 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762433 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762437 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762440 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762443 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762446 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762449 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762452 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 00:06:30.771866 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762455 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762458 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762460 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762463 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762466 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762469 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762472 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762475 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762477 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762484 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762487 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762490 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762493 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762497 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762500 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762503 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762505 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762508 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762511 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762514 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 00:06:30.772412 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762517 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762520 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762522 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762525 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762530 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762533 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762536 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762539 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762541 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762544 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762547 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762550 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762553 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762555 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762558 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762561 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762564 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762566 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762569 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762572 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 00:06:30.772945 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762575 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762578 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762586 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762589 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762592 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762596 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762600 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762604 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762609 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762613 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762616 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762619 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762622 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762625 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762628 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762631 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762635 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762638 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762641 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 00:06:30.773538 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762643 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762646 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762649 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762653 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762657 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762660 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762663 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762666 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762669 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762672 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762675 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762678 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762681 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762683 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762686 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762689 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762692 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762695 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 00:06:30.774044 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.762699 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.763424 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.770829 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.770860 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.771643 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.771652 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771722 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771730 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771733 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771737 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771742 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771745 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771750 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771755 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 00:06:30.774530 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771758 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771762 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771767 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771770 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771773 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771776 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771779 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771782 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771785 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771788 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771791 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771794 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771797 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771800 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771803 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771806 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771809 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771812 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771814 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771817 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 00:06:30.774922 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771820 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771825 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771828 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771831 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771834 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771837 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771840 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771843 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771846 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771849 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771852 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771854 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771857 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771860 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771863 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771866 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771868 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771871 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771874 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771878 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 00:06:30.775541 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771881 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771884 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771887 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771890 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771892 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771895 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771898 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771901 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771903 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771907 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771909 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771912 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771915 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771919 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771923 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771926 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771929 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771932 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771936 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 00:06:30.776086 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771939 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771942 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771944 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771947 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771951 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771956 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771959 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771962 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771965 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771967 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771970 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771973 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771976 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771979 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771984 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771987 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771991 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771994 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.771997 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 00:06:30.776629 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772001 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772004 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772007 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772010 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772013 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772016 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772019 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772023 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772026 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772029 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772032 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772035 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772037 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.772043 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772189 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 00:06:30.777165 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772197 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772201 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772205 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772208 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772211 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772214 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772217 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772221 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772225 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772228 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772231 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772234 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772236 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772239 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772241 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772244 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772247 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772249 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772254 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 00:06:30.777601 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772258 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772261 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772264 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772267 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772270 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772273 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772276 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772279 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772282 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772285 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772288 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772291 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772293 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772296 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772299 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772302 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772305 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772307 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772310 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772313 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 00:06:30.778113 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772316 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772318 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772321 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772324 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772326 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772329 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772332 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772334 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772337 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772340 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772342 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772345 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772348 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772351 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772354 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772357 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772359 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772363 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772365 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772368 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772371 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 00:06:30.778662 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772374 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772377 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772380 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772382 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772385 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772388 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772391 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772393 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772396 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772399 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772401 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772404 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772407 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772409 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772412 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772415 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772418 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772421 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772423 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772426 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 00:06:30.779239 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772429 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772432 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772435 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772438 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772441 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772444 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772447 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772450 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772453 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772456 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772458 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772461 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772464 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772467 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772471 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772474 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772477 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 00:06:30.779764 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:30.772481 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 00:06:30.780367 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.772487 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 00:06:30.780367 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.772636 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 00:06:30.780367 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.775793 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 00:06:30.780367 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.777410 2575 server.go:1019] "Starting client certificate rotation" Jul 31 00:06:30.780367 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.777528 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 00:06:30.780367 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.777614 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 00:06:30.781116 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.781095 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 00:06:30.789496 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.789469 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 00:06:30.803502 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.803474 2575 log.go:25] "Validated CRI v1 runtime API" Jul 31 00:06:30.805967 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.805947 2575 log.go:25] "Validated CRI v1 image API" Jul 31 00:06:30.806371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.806350 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 00:06:30.807987 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.807963 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 00:06:30.810635 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.810602 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7cc0c122-0d1a-4d3e-aa46-c57b414193e6:/dev/nvme0n1p4 86f282ca-2e2b-4093-90e0-8d3a98a695ac:/dev/nvme0n1p3] Jul 31 00:06:30.810703 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.810633 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 00:06:30.816728 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.816593 2575 manager.go:216] Machine: {Timestamp:2026-07-31 00:06:30.815167621 +0000 UTC m=+0.338135360 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102732 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27f7609a03ed6fb9b81439798b1df6 SystemUUID:ec27f760-9a03-ed6f-b9b8-1439798b1df6 BootID:70fafaa4-40c7-4105-b3ca-0c2420b83e1a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:34:50:e1:e3:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:34:50:e1:e3:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:9e:d0:a1:58:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 00:06:30.816728 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.816723 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 00:06:30.816882 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.816870 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 00:06:30.820582 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.820548 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 00:06:30.820758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.820584 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 00:06:30.821900 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.821887 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 00:06:30.821946 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.821902 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 00:06:30.821946 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.821911 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 00:06:30.822553 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.822540 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 00:06:30.823315 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.823303 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 00:06:30.827322 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.827301 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kqm95" Jul 31 00:06:30.827378 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.827361 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 31 00:06:30.827556 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.827543 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 00:06:30.827593 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.827577 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 31 00:06:30.827593 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.827589 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 00:06:30.827655 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.827603 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 00:06:30.827655 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.827616 2575 kubelet.go:398] "Adding apiserver pod source" Jul 31 00:06:30.827655 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.827626 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 00:06:30.828022 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.828008 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 00:06:30.828098 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.828031 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 00:06:30.830459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.830437 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 00:06:30.831932 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.831915 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 00:06:30.831985 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.831950 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 00:06:30.832552 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.832536 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kqm95" Jul 31 00:06:30.833343 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833315 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 00:06:30.833343 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833336 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 00:06:30.833343 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833348 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833357 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833370 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833376 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833385 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833391 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833400 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833406 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833415 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 00:06:30.833504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.833421 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 00:06:30.834309 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.834299 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 00:06:30.834309 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.834309 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 00:06:30.838920 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.838891 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 00:06:30.838920 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.838904 2575 server.go:1268] "Started kubelet" Jul 31 00:06:30.839110 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.838943 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 00:06:30.839110 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:30.839040 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 00:06:30.839110 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.839053 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 00:06:30.839279 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:30.839124 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 00:06:30.839279 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.839184 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 00:06:30.840080 ip-10-0-134-191 systemd[1]: Started Kubernetes Kubelet. Jul 31 00:06:30.840329 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.840310 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 00:06:30.844366 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.844344 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 31 00:06:30.845645 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:30.845623 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 00:06:30.849424 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.849400 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 00:06:30.849424 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.849407 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 00:06:30.849616 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:30.849502 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:30.850156 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.850064 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 00:06:30.850156 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.850081 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 00:06:30.850156 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.850060 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 00:06:30.850367 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.850278 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 31 00:06:30.850367 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.850288 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 31 00:06:30.850967 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.850932 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 00:06:30.851039 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.850971 2575 factory.go:55] Registering systemd factory Jul 31 00:06:30.851039 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.850982 2575 factory.go:223] Registration of the systemd container factory successfully Jul 31 00:06:30.851294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.851273 2575 factory.go:153] Registering CRI-O factory Jul 31 00:06:30.851294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.851296 2575 factory.go:223] Registration of the crio container factory successfully Jul 31 00:06:30.851426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.851317 2575 factory.go:103] Registering Raw factory Jul 31 00:06:30.851426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.851330 2575 manager.go:1196] Started watching for new ooms in manager Jul 31 00:06:30.851827 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.851773 2575 manager.go:349] Starting recovery of all containers Jul 31 00:06:30.852257 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.852231 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 00:06:30.856156 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:30.856120 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-191.ec2.internal\" not found" node="ip-10-0-134-191.ec2.internal" Jul 31 00:06:30.863493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.863476 2575 manager.go:354] Recovery completed Jul 31 00:06:30.869068 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.869049 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 00:06:30.873768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.873737 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientMemory" Jul 31 00:06:30.873856 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.873788 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 00:06:30.873856 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.873804 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientPID" Jul 31 00:06:30.874509 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.874494 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 00:06:30.874509 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.874507 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 00:06:30.874633 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.874531 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 31 00:06:30.874633 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.874557 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 00:06:30.875891 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.875879 2575 policy_none.go:49] "None policy: Start" Jul 31 00:06:30.875929 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.875895 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 00:06:30.875929 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.875907 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 00:06:30.876868 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.876858 2575 policy_none.go:47] "Start" Jul 31 00:06:30.884533 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.884515 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientMemory" Jul 31 00:06:30.884626 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.884547 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 00:06:30.884626 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.884563 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientPID" Jul 31 00:06:30.915046 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.914858 2575 manager.go:337] "Starting Device Plugin manager" Jul 31 00:06:30.915196 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:30.915071 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 00:06:30.915196 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.915085 2575 server.go:87] "Starting device plugin registration server" Jul 31 00:06:30.915416 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.915402 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 00:06:30.915464 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.915421 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 00:06:30.915545 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.915526 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 00:06:30.915623 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.915616 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 00:06:30.915669 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:30.915626 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 00:06:30.916187 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:30.916144 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 00:06:30.916285 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:30.916222 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.017051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.016949 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 00:06:31.018293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.018025 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientMemory" Jul 31 00:06:31.018293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.018062 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 00:06:31.018293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.018078 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientPID" Jul 31 00:06:31.018293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.018109 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.024620 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.024599 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.024698 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.024629 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-191.ec2.internal\": node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.037712 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.037679 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.040760 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.040719 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 00:06:31.042051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.042029 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 00:06:31.042177 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.042069 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 00:06:31.042177 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.042096 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 00:06:31.042177 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.042152 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 00:06:31.044387 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.044360 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 00:06:31.143723 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.143675 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.143723 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.143693 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal"] Jul 31 00:06:31.143934 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.143796 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 00:06:31.145593 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.145574 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientMemory" Jul 31 00:06:31.145713 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.145603 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 00:06:31.145713 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.145614 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientPID" Jul 31 00:06:31.146923 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.146910 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 00:06:31.147049 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.147028 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.147086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.147061 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 00:06:31.147707 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.147691 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientMemory" Jul 31 00:06:31.147707 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.147702 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientMemory" Jul 31 00:06:31.147814 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.147721 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 00:06:31.147814 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.147724 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 00:06:31.147814 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.147737 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientPID" Jul 31 00:06:31.147814 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.147749 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientPID" Jul 31 00:06:31.149040 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.149023 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.149145 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.149050 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 00:06:31.149834 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.149818 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientMemory" Jul 31 00:06:31.149911 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.149846 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 00:06:31.149911 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.149859 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeHasSufficientPID" Jul 31 00:06:31.177238 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.177203 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-191.ec2.internal\" not found" node="ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.182206 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.182173 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-191.ec2.internal\" not found" node="ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.244289 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.244256 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.251870 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.251841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/637e762b63558847f949d39093da072e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal\" (UID: \"637e762b63558847f949d39093da072e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.251962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.251875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/637e762b63558847f949d39093da072e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal\" (UID: \"637e762b63558847f949d39093da072e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.251962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.251894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3820b175871ebd792360ac3026b04d32-config\") pod \"kube-apiserver-proxy-ip-10-0-134-191.ec2.internal\" (UID: \"3820b175871ebd792360ac3026b04d32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.344813 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.344727 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.352618 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.352586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/637e762b63558847f949d39093da072e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal\" (UID: \"637e762b63558847f949d39093da072e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.352702 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.352630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/637e762b63558847f949d39093da072e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal\" (UID: \"637e762b63558847f949d39093da072e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.352702 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.352648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3820b175871ebd792360ac3026b04d32-config\") pod \"kube-apiserver-proxy-ip-10-0-134-191.ec2.internal\" (UID: \"3820b175871ebd792360ac3026b04d32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.352702 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.352697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/637e762b63558847f949d39093da072e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal\" (UID: \"637e762b63558847f949d39093da072e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.352799 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.352705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/637e762b63558847f949d39093da072e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal\" (UID: \"637e762b63558847f949d39093da072e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.352799 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.352758 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3820b175871ebd792360ac3026b04d32-config\") pod \"kube-apiserver-proxy-ip-10-0-134-191.ec2.internal\" (UID: \"3820b175871ebd792360ac3026b04d32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.451754 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.451712 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.480997 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.480963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.486113 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.485626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.551969 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.551932 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.655518 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.655430 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-191.ec2.internal\" not found" Jul 31 00:06:31.733293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.733269 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 00:06:31.750744 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.750713 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.763153 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.763114 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 00:06:31.765309 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.765292 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" Jul 31 00:06:31.773380 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.773351 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 00:06:31.776730 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.776711 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 00:06:31.776877 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.776858 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 00:06:31.776936 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.776881 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 00:06:31.776936 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.776886 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 00:06:31.828322 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.828289 2575 apiserver.go:52] "Watching apiserver" Jul 31 00:06:31.837286 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.837253 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 00:01:30 +0000 UTC" deadline="2028-04-05 07:34:53.912641175 +0000 UTC" Jul 31 00:06:31.837286 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.837280 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14743h28m22.075364048s" Jul 31 00:06:31.847128 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.847084 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 00:06:31.847439 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.847417 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal","openshift-multus/network-metrics-daemon-6xpcj","openshift-ovn-kubernetes/ovnkube-node-gflzn","kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vqx58","openshift-image-registry/node-ca-9ck7r","openshift-multus/multus-additional-cni-plugins-wtk76","openshift-multus/multus-wt2nl","openshift-network-diagnostics/network-check-target-s2ksf","openshift-network-operator/iptables-alerter-bsbtr","kube-system/global-pull-secret-syncer-gjqf5","kube-system/konnectivity-agent-6cvnn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp","openshift-dns/node-resolver-5f6lz"] Jul 31 00:06:31.849128 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.849096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:31.849224 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.849200 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:31.850218 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.850201 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 00:06:31.850379 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.850362 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.851522 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.851502 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.852712 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.852694 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:31.853262 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.853245 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 00:06:31.853892 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.853864 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 00:06:31.854133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.853914 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 00:06:31.854133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.853973 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 00:06:31.854133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854009 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 00:06:31.854133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854029 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lmrn9\"" Jul 31 00:06:31.854133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854010 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 00:06:31.854370 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.854633 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854616 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 00:06:31.854692 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854619 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 00:06:31.854692 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854646 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jnpvm\"" Jul 31 00:06:31.854903 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854887 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-26w5c\"" Jul 31 00:06:31.854951 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854907 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 00:06:31.855000 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.854890 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 00:06:31.855187 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.855172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.855247 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.855211 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 00:06:31.856039 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8f4\" (UniqueName: \"kubernetes.io/projected/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-kube-api-access-2c8f4\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:31.856172 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-systemd-units\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856172 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-etc-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856172 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovnkube-script-lib\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856172 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxdr\" (UniqueName: \"kubernetes.io/projected/b0967a28-06e3-4d79-9e5e-930d0d9c2152-kube-api-access-kkxdr\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856172 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-slash\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-run-netns\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-systemd\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-ovn\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-log-socket\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-run-ovn-kubernetes\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-kubelet\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-node-log\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856434 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-cni-bin\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856462 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovnkube-config\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-env-overrides\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856533 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.856499 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-var-lib-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.856636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-cni-netd\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.857044 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovn-node-metrics-cert\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.857044 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856683 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 00:06:31.857044 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.856689 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 00:06:31.857174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.857066 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 00:06:31.857174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.857098 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 00:06:31.857737 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.857679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:31.857813 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.857784 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 00:06:31.857986 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.857968 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wxx7z\"" Jul 31 00:06:31.858063 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.857974 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r7hsc\"" Jul 31 00:06:31.858975 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.858956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:31.859049 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.859035 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:31.859823 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.859808 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 00:06:31.859887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.859809 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5znz2\"" Jul 31 00:06:31.859986 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.859973 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 00:06:31.860178 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.860161 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 00:06:31.860898 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.860884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:31.862224 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.862207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:31.863426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.863404 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 00:06:31.864315 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.863732 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 00:06:31.864315 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.863922 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:31.865233 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.864442 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c2k8w\"" Jul 31 00:06:31.865233 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.864831 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jffn5\"" Jul 31 00:06:31.865811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.865524 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 00:06:31.865811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.865608 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 00:06:31.865811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.865607 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 00:06:31.866595 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.866579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nxwms\"" Jul 31 00:06:31.866698 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.866609 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 00:06:31.866698 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.866647 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 00:06:31.866788 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.866703 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 00:06:31.878689 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.878669 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 00:06:31.885731 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.885709 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7jr7" Jul 31 00:06:31.894594 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.894559 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7jr7" Jul 31 00:06:31.950982 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.950956 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 00:06:31.957086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-registration-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:31.957086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysctl-d\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.957298 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028bd031-00ad-4234-bb3d-fef11b3ff915-tmp\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.957298 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fccb5b35-15ad-426c-a0c5-83eb68df8166-cni-binary-copy\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.957298 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wlq\" (UniqueName: \"kubernetes.io/projected/fccb5b35-15ad-426c-a0c5-83eb68df8166-kube-api-access-h6wlq\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.957298 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:31.957298 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-tuned\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.957479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbsc\" (UniqueName: \"kubernetes.io/projected/028bd031-00ad-4234-bb3d-fef11b3ff915-kube-api-access-mpbsc\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.957479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-dbus\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:31.957479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-systemd\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.957479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-cni-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.957479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-cni-bin\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.957479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-multus-certs\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.957479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9ll\" (UniqueName: \"kubernetes.io/projected/814f97fa-4c99-483b-a086-c3da15df60a6-kube-api-access-wd9ll\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:31.957725 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.957462 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:31.957725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.957725 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:31.957567 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:06:32.457533738 +0000 UTC m=+1.980501461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:31.957725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-systemd\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.957725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-run-ovn-kubernetes\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.957725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovnkube-config\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957735 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cnibin\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-systemd\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-run-ovn-kubernetes\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-node-log\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-kubelet-config\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/afbe38cc-74f5-4daf-a158-120a7a583821-agent-certs\") pod \"konnectivity-agent-6cvnn\" (UID: \"afbe38cc-74f5-4daf-a158-120a7a583821\") " pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-node-log\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-device-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:31.957960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814f97fa-4c99-483b-a086-c3da15df60a6-tmp-dir\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.957977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-sys-fs\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2n54\" (UniqueName: \"kubernetes.io/projected/b493d74f-267f-4463-b67d-0df930eba2ee-kube-api-access-g2n54\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-modprobe-d\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysctl-conf\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-k8s-cni-cncf-io\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-systemd-units\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-etc-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovnkube-script-lib\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958232 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-etc-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxdr\" (UniqueName: \"kubernetes.io/projected/b0967a28-06e3-4d79-9e5e-930d0d9c2152-kube-api-access-kkxdr\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-systemd-units\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/232e2b98-3c4e-415e-b538-6e350a6df518-serviceca\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:31.958470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-run-netns\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-run-netns\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-env-overrides\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-host-slash\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-lib-modules\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-cnibin\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-hostroot\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/232e2b98-3c4e-415e-b538-6e350a6df518-host\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958748 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-cni-bin\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958801 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/afbe38cc-74f5-4daf-a158-120a7a583821-konnectivity-ca\") pod \"konnectivity-agent-6cvnn\" (UID: \"afbe38cc-74f5-4daf-a158-120a7a583821\") " pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysconfig\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-host\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-kubelet\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958906 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-cni-bin\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-conf-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.959291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.958981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-etc-kubernetes\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-var-lib-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-socket-dir-parent\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-netns\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959135 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-var-lib-openvswitch\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-cni-multus\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovnkube-config\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-env-overrides\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovnkube-script-lib\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdtxr\" (UniqueName: \"kubernetes.io/projected/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-kube-api-access-zdtxr\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8f4\" (UniqueName: \"kubernetes.io/projected/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-kube-api-access-2c8f4\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-iptables-alerter-script\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-kubernetes\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-daemon-config\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:31.960025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-slash\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxgb2\" (UniqueName: \"kubernetes.io/projected/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-kube-api-access-gxgb2\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959801 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-slash\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-var-lib-kubelet\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-kubelet\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-run\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959942 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-kubelet\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbbn\" (UniqueName: \"kubernetes.io/projected/232e2b98-3c4e-415e-b538-6e350a6df518-kube-api-access-2bbbn\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.959996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-cni-netd\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovn-node-metrics-cert\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-sys\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960077 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-host-cni-netd\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-os-release\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-os-release\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.960833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-ovn\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960380 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-run-ovn\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-log-socket\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-socket-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-system-cni-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/814f97fa-4c99-483b-a086-c3da15df60a6-hosts-file\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:31.961389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.960800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0967a28-06e3-4d79-9e5e-930d0d9c2152-log-socket\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.963839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.963816 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0967a28-06e3-4d79-9e5e-930d0d9c2152-ovn-node-metrics-cert\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.969291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.969257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxdr\" (UniqueName: \"kubernetes.io/projected/b0967a28-06e3-4d79-9e5e-930d0d9c2152-kube-api-access-kkxdr\") pod \"ovnkube-node-gflzn\" (UID: \"b0967a28-06e3-4d79-9e5e-930d0d9c2152\") " pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:31.969618 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:31.969597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8f4\" (UniqueName: \"kubernetes.io/projected/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-kube-api-access-2c8f4\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:32.061070 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:32.061070 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-sys\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-os-release\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-os-release\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-socket-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061175 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-sys\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-system-cni-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.061214 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/814f97fa-4c99-483b-a086-c3da15df60a6-hosts-file\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-registration-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-os-release\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-system-cni-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.061290 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret podName:e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:32.561268963 +0000 UTC m=+2.084236687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret") pod "global-pull-secret-syncer-gjqf5" (UID: "e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9") : object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:32.061294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061301 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-registration-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061315 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-os-release\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061322 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/814f97fa-4c99-483b-a086-c3da15df60a6-hosts-file\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-socket-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysctl-d\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028bd031-00ad-4234-bb3d-fef11b3ff915-tmp\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fccb5b35-15ad-426c-a0c5-83eb68df8166-cni-binary-copy\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wlq\" (UniqueName: \"kubernetes.io/projected/fccb5b35-15ad-426c-a0c5-83eb68df8166-kube-api-access-h6wlq\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-tuned\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysctl-d\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbsc\" (UniqueName: \"kubernetes.io/projected/028bd031-00ad-4234-bb3d-fef11b3ff915-kube-api-access-mpbsc\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-dbus\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-systemd\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-cni-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-cni-bin\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-multus-certs\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9ll\" (UniqueName: \"kubernetes.io/projected/814f97fa-4c99-483b-a086-c3da15df60a6-kube-api-access-wd9ll\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:32.061919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061775 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-dbus\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061782 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-cni-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061713 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-cni-bin\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cnibin\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061814 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-multus-certs\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cnibin\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-systemd\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061876 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061929 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-kubelet-config\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/afbe38cc-74f5-4daf-a158-120a7a583821-agent-certs\") pod \"konnectivity-agent-6cvnn\" (UID: \"afbe38cc-74f5-4daf-a158-120a7a583821\") " pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.061975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-device-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814f97fa-4c99-483b-a086-c3da15df60a6-tmp-dir\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-kubelet-config\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:32.062818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-sys-fs\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062051 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fccb5b35-15ad-426c-a0c5-83eb68df8166-cni-binary-copy\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2n54\" (UniqueName: \"kubernetes.io/projected/b493d74f-267f-4463-b67d-0df930eba2ee-kube-api-access-g2n54\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-device-dir\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-modprobe-d\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b493d74f-267f-4463-b67d-0df930eba2ee-sys-fs\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysctl-conf\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-k8s-cni-cncf-io\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/232e2b98-3c4e-415e-b538-6e350a6df518-serviceca\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-host-slash\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-lib-modules\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-cnibin\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-hostroot\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/232e2b98-3c4e-415e-b538-6e350a6df518-host\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062403 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814f97fa-4c99-483b-a086-c3da15df60a6-tmp-dir\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/afbe38cc-74f5-4daf-a158-120a7a583821-konnectivity-ca\") pod \"konnectivity-agent-6cvnn\" (UID: \"afbe38cc-74f5-4daf-a158-120a7a583821\") " pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:32.063637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysconfig\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062490 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-host-slash\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-host\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-kubelet\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062541 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/232e2b98-3c4e-415e-b538-6e350a6df518-host\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-conf-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-etc-kubernetes\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-socket-dir-parent\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062500 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-hostroot\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062661 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-netns\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-cni-multus\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysconfig\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdtxr\" (UniqueName: \"kubernetes.io/projected/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-kube-api-access-zdtxr\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-sysctl-conf\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062764 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-host\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-iptables-alerter-script\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:32.064500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062774 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/232e2b98-3c4e-415e-b538-6e350a6df518-serviceca\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-kubernetes\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062812 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-k8s-cni-cncf-io\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-daemon-config\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-modprobe-d\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-run-netns\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062684 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-cnibin\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-kubelet\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062938 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-conf-dir\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-etc-kubernetes\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-socket-dir-parent\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxgb2\" (UniqueName: \"kubernetes.io/projected/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-kube-api-access-gxgb2\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-var-lib-kubelet\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.062636 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-lib-modules\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-run\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbbn\" (UniqueName: \"kubernetes.io/projected/232e2b98-3c4e-415e-b538-6e350a6df518-kube-api-access-2bbbn\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:32.065348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063202 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-run\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/afbe38cc-74f5-4daf-a158-120a7a583821-konnectivity-ca\") pod \"konnectivity-agent-6cvnn\" (UID: \"afbe38cc-74f5-4daf-a158-120a7a583821\") " pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-var-lib-kubelet\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063278 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-kubernetes\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fccb5b35-15ad-426c-a0c5-83eb68df8166-host-var-lib-cni-multus\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063412 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fccb5b35-15ad-426c-a0c5-83eb68df8166-multus-daemon-config\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063547 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063764 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-iptables-alerter-script\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.063998 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028bd031-00ad-4234-bb3d-fef11b3ff915-tmp\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.064770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/afbe38cc-74f5-4daf-a158-120a7a583821-agent-certs\") pod \"konnectivity-agent-6cvnn\" (UID: \"afbe38cc-74f5-4daf-a158-120a7a583821\") " pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:32.065846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.065423 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/028bd031-00ad-4234-bb3d-fef11b3ff915-etc-tuned\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.068527 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.068503 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 00:06:32.068527 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.068529 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 00:06:32.068713 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.068542 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jqrtz for pod openshift-network-diagnostics/network-check-target-s2ksf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:32.068713 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.068660 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz podName:65a77142-1094-493e-862a-3a0a9fafe475 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:32.568639963 +0000 UTC m=+2.091607713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jqrtz" (UniqueName: "kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz") pod "network-check-target-s2ksf" (UID: "65a77142-1094-493e-862a-3a0a9fafe475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:32.070405 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.070375 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbsc\" (UniqueName: \"kubernetes.io/projected/028bd031-00ad-4234-bb3d-fef11b3ff915-kube-api-access-mpbsc\") pod \"tuned-vqx58\" (UID: \"028bd031-00ad-4234-bb3d-fef11b3ff915\") " pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.070551 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.070529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wlq\" (UniqueName: \"kubernetes.io/projected/fccb5b35-15ad-426c-a0c5-83eb68df8166-kube-api-access-h6wlq\") pod \"multus-wt2nl\" (UID: \"fccb5b35-15ad-426c-a0c5-83eb68df8166\") " pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.070913 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.070890 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9ll\" (UniqueName: \"kubernetes.io/projected/814f97fa-4c99-483b-a086-c3da15df60a6-kube-api-access-wd9ll\") pod \"node-resolver-5f6lz\" (UID: \"814f97fa-4c99-483b-a086-c3da15df60a6\") " pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:32.071140 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.071108 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2n54\" (UniqueName: \"kubernetes.io/projected/b493d74f-267f-4463-b67d-0df930eba2ee-kube-api-access-g2n54\") pod \"aws-ebs-csi-driver-node-r4dbp\" (UID: \"b493d74f-267f-4463-b67d-0df930eba2ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.071382 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.071358 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbbn\" (UniqueName: \"kubernetes.io/projected/232e2b98-3c4e-415e-b538-6e350a6df518-kube-api-access-2bbbn\") pod \"node-ca-9ck7r\" (UID: \"232e2b98-3c4e-415e-b538-6e350a6df518\") " pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:32.071610 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.071580 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxgb2\" (UniqueName: \"kubernetes.io/projected/e470d8d6-7673-43d1-a5d3-ccf49c41c6bb-kube-api-access-gxgb2\") pod \"iptables-alerter-bsbtr\" (UID: \"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb\") " pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:32.071847 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.071833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdtxr\" (UniqueName: \"kubernetes.io/projected/1f877c84-ab31-4503-9fcd-bb0f621e4ef9-kube-api-access-zdtxr\") pod \"multus-additional-cni-plugins-wtk76\" (UID: \"1f877c84-ab31-4503-9fcd-bb0f621e4ef9\") " pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.194156 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.194034 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:32.201418 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.201378 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vqx58" Jul 31 00:06:32.205236 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.205214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ck7r" Jul 31 00:06:32.227141 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.227095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wtk76" Jul 31 00:06:32.241716 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.241685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wt2nl" Jul 31 00:06:32.259683 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.259654 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bsbtr" Jul 31 00:06:32.278392 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.278360 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:32.294401 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.294362 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" Jul 31 00:06:32.298424 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.298400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5f6lz" Jul 31 00:06:32.423317 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.423271 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637e762b63558847f949d39093da072e.slice/crio-e9c398ad11fcc19a8a3b2ab002537215ba667746a56d45ee89ce130c700b8cff WatchSource:0}: Error finding container e9c398ad11fcc19a8a3b2ab002537215ba667746a56d45ee89ce130c700b8cff: Status 404 returned error can't find the container with id e9c398ad11fcc19a8a3b2ab002537215ba667746a56d45ee89ce130c700b8cff Jul 31 00:06:32.423758 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.423736 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3820b175871ebd792360ac3026b04d32.slice/crio-c0216af41d6e25edfb01d969a5b4465c2c567729f3194a512a0aa4a28be7bdf6 WatchSource:0}: Error finding container c0216af41d6e25edfb01d969a5b4465c2c567729f3194a512a0aa4a28be7bdf6: Status 404 returned error can't find the container with id c0216af41d6e25edfb01d969a5b4465c2c567729f3194a512a0aa4a28be7bdf6 Jul 31 00:06:32.428418 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.428400 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 00:06:32.465244 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.465147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:32.465387 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.465329 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:32.465446 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.465402 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:06:33.465385018 +0000 UTC m=+2.988352760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:32.529759 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.529707 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccb5b35_15ad_426c_a0c5_83eb68df8166.slice/crio-3fd4c5227874d563ab69a5cc56f57f3e947f8408ba90e0f44016f7ece355a1bf WatchSource:0}: Error finding container 3fd4c5227874d563ab69a5cc56f57f3e947f8408ba90e0f44016f7ece355a1bf: Status 404 returned error can't find the container with id 3fd4c5227874d563ab69a5cc56f57f3e947f8408ba90e0f44016f7ece355a1bf Jul 31 00:06:32.564520 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.564492 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0967a28_06e3_4d79_9e5e_930d0d9c2152.slice/crio-b14394551006e60a5ad689cf4ce8010e98e9c3cb4898f76f8f7713103466ab13 WatchSource:0}: Error finding container b14394551006e60a5ad689cf4ce8010e98e9c3cb4898f76f8f7713103466ab13: Status 404 returned error can't find the container with id b14394551006e60a5ad689cf4ce8010e98e9c3cb4898f76f8f7713103466ab13 Jul 31 00:06:32.565625 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.565599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:32.565747 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.565734 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:32.565804 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.565789 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret podName:e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:33.565771747 +0000 UTC m=+3.088739473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret") pod "global-pull-secret-syncer-gjqf5" (UID: "e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9") : object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:32.625512 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.625484 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028bd031_00ad_4234_bb3d_fef11b3ff915.slice/crio-a5ec643b52e2076b624c04c219c7a73dad8279220170fd3578b9c3c0816bc15d WatchSource:0}: Error finding container a5ec643b52e2076b624c04c219c7a73dad8279220170fd3578b9c3c0816bc15d: Status 404 returned error can't find the container with id a5ec643b52e2076b624c04c219c7a73dad8279220170fd3578b9c3c0816bc15d Jul 31 00:06:32.666377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.666342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:32.666549 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.666472 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 00:06:32.666549 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.666491 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 00:06:32.666549 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.666502 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jqrtz for pod openshift-network-diagnostics/network-check-target-s2ksf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:32.666664 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:32.666553 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz podName:65a77142-1094-493e-862a-3a0a9fafe475 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:33.666539399 +0000 UTC m=+3.189507136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqrtz" (UniqueName: "kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz") pod "network-check-target-s2ksf" (UID: "65a77142-1094-493e-862a-3a0a9fafe475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:32.680314 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.680282 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f877c84_ab31_4503_9fcd_bb0f621e4ef9.slice/crio-23f368d385d438b15d0f2c8cabf59d153f91ffbf7e505e228142d4b80a6710da WatchSource:0}: Error finding container 23f368d385d438b15d0f2c8cabf59d153f91ffbf7e505e228142d4b80a6710da: Status 404 returned error can't find the container with id 23f368d385d438b15d0f2c8cabf59d153f91ffbf7e505e228142d4b80a6710da Jul 31 00:06:32.680679 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.680649 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb493d74f_267f_4463_b67d_0df930eba2ee.slice/crio-07dc1cb15e457ab3cc085b2425607d6d6a64f8868b0fa3875d4355f2eb4bc9e8 WatchSource:0}: Error finding container 07dc1cb15e457ab3cc085b2425607d6d6a64f8868b0fa3875d4355f2eb4bc9e8: Status 404 returned error can't find the container with id 07dc1cb15e457ab3cc085b2425607d6d6a64f8868b0fa3875d4355f2eb4bc9e8 Jul 31 00:06:32.704895 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.704867 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbe38cc_74f5_4daf_a158_120a7a583821.slice/crio-4bb151a408b8ccec24424ab86116918be65c4c6d345ca01a948f0727e87818d2 WatchSource:0}: Error finding container 4bb151a408b8ccec24424ab86116918be65c4c6d345ca01a948f0727e87818d2: Status 404 returned error can't find the container with id 4bb151a408b8ccec24424ab86116918be65c4c6d345ca01a948f0727e87818d2 Jul 31 00:06:32.787195 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.787162 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814f97fa_4c99_483b_a086_c3da15df60a6.slice/crio-4ce16624cd24e4d3c38f1682a676fa5a03fddb9fd7a2bca4120887cf0d1e858c WatchSource:0}: Error finding container 4ce16624cd24e4d3c38f1682a676fa5a03fddb9fd7a2bca4120887cf0d1e858c: Status 404 returned error can't find the container with id 4ce16624cd24e4d3c38f1682a676fa5a03fddb9fd7a2bca4120887cf0d1e858c Jul 31 00:06:32.795266 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.795243 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 00:06:32.896790 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.896728 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 00:01:31 +0000 UTC" deadline="2028-01-22 22:09:11.758296233 +0000 UTC" Jul 31 00:06:32.896790 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:32.896785 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12982h2m38.86151447s" Jul 31 00:06:32.955170 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:32.955128 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode470d8d6_7673_43d1_a5d3_ccf49c41c6bb.slice/crio-b568297c736efda96eb83555f40d74ce9fc5d0786aaf3f4cf30f0ca5524b58ce WatchSource:0}: Error finding container b568297c736efda96eb83555f40d74ce9fc5d0786aaf3f4cf30f0ca5524b58ce: Status 404 returned error can't find the container with id b568297c736efda96eb83555f40d74ce9fc5d0786aaf3f4cf30f0ca5524b58ce Jul 31 00:06:33.003817 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:06:33.003781 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232e2b98_3c4e_415e_b538_6e350a6df518.slice/crio-1246da250a3b46a9b07e2a0bf577a51a7f4d9e594f9937fbd1cc9fa99fab20f7 WatchSource:0}: Error finding container 1246da250a3b46a9b07e2a0bf577a51a7f4d9e594f9937fbd1cc9fa99fab20f7: Status 404 returned error can't find the container with id 1246da250a3b46a9b07e2a0bf577a51a7f4d9e594f9937fbd1cc9fa99fab20f7 Jul 31 00:06:33.007489 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.007465 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 00:06:33.048555 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.048448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ck7r" event={"ID":"232e2b98-3c4e-415e-b538-6e350a6df518","Type":"ContainerStarted","Data":"1246da250a3b46a9b07e2a0bf577a51a7f4d9e594f9937fbd1cc9fa99fab20f7"} Jul 31 00:06:33.049992 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.049961 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6cvnn" event={"ID":"afbe38cc-74f5-4daf-a158-120a7a583821","Type":"ContainerStarted","Data":"4bb151a408b8ccec24424ab86116918be65c4c6d345ca01a948f0727e87818d2"} Jul 31 00:06:33.051327 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.051301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" event={"ID":"b493d74f-267f-4463-b67d-0df930eba2ee","Type":"ContainerStarted","Data":"07dc1cb15e457ab3cc085b2425607d6d6a64f8868b0fa3875d4355f2eb4bc9e8"} Jul 31 00:06:33.052618 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.052593 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtk76" event={"ID":"1f877c84-ab31-4503-9fcd-bb0f621e4ef9","Type":"ContainerStarted","Data":"23f368d385d438b15d0f2c8cabf59d153f91ffbf7e505e228142d4b80a6710da"} Jul 31 00:06:33.054221 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.054194 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vqx58" event={"ID":"028bd031-00ad-4234-bb3d-fef11b3ff915","Type":"ContainerStarted","Data":"a5ec643b52e2076b624c04c219c7a73dad8279220170fd3578b9c3c0816bc15d"} Jul 31 00:06:33.055713 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.055687 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"b14394551006e60a5ad689cf4ce8010e98e9c3cb4898f76f8f7713103466ab13"} Jul 31 00:06:33.058272 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.058246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wt2nl" event={"ID":"fccb5b35-15ad-426c-a0c5-83eb68df8166","Type":"ContainerStarted","Data":"3fd4c5227874d563ab69a5cc56f57f3e947f8408ba90e0f44016f7ece355a1bf"} Jul 31 00:06:33.060316 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.060251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" event={"ID":"637e762b63558847f949d39093da072e","Type":"ContainerStarted","Data":"e9c398ad11fcc19a8a3b2ab002537215ba667746a56d45ee89ce130c700b8cff"} Jul 31 00:06:33.062038 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.061987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bsbtr" event={"ID":"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb","Type":"ContainerStarted","Data":"b568297c736efda96eb83555f40d74ce9fc5d0786aaf3f4cf30f0ca5524b58ce"} Jul 31 00:06:33.064080 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.064057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5f6lz" event={"ID":"814f97fa-4c99-483b-a086-c3da15df60a6","Type":"ContainerStarted","Data":"4ce16624cd24e4d3c38f1682a676fa5a03fddb9fd7a2bca4120887cf0d1e858c"} Jul 31 00:06:33.065872 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.065323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" event={"ID":"3820b175871ebd792360ac3026b04d32","Type":"ContainerStarted","Data":"c0216af41d6e25edfb01d969a5b4465c2c567729f3194a512a0aa4a28be7bdf6"} Jul 31 00:06:33.472493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.472409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:33.472666 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:33.472596 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:33.472666 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:33.472660 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:06:35.472641774 +0000 UTC m=+4.995609500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:33.573931 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.573376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:33.573931 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:33.573517 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:33.573931 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:33.573585 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret podName:e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:35.573565508 +0000 UTC m=+5.096533232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret") pod "global-pull-secret-syncer-gjqf5" (UID: "e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9") : object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:33.674343 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.674290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:33.674547 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:33.674529 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 00:06:33.674638 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:33.674558 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 00:06:33.674638 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:33.674571 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jqrtz for pod openshift-network-diagnostics/network-check-target-s2ksf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:33.674749 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:33.674641 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz podName:65a77142-1094-493e-862a-3a0a9fafe475 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:35.674622783 +0000 UTC m=+5.197590511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqrtz" (UniqueName: "kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz") pod "network-check-target-s2ksf" (UID: "65a77142-1094-493e-862a-3a0a9fafe475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:33.897151 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.897007 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 00:01:31 +0000 UTC" deadline="2028-03-10 21:25:02.109312696 +0000 UTC" Jul 31 00:06:33.897151 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:33.897048 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14133h18m28.212269696s" Jul 31 00:06:34.043023 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:34.042512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:34.043023 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:34.042544 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:34.043023 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:34.042662 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:34.043023 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:34.042786 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:34.043023 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:34.042847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:34.043023 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:34.042921 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:34.775378 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:34.775337 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 00:06:35.493213 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:35.493173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:35.493711 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:35.493336 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:35.493711 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:35.493407 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:06:39.493386916 +0000 UTC m=+9.016354641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:35.594304 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:35.594266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:35.601445 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:35.601407 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:35.601634 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:35.601533 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret podName:e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:39.601489189 +0000 UTC m=+9.124456914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret") pod "global-pull-secret-syncer-gjqf5" (UID: "e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9") : object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:35.695540 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:35.695500 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:35.695726 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:35.695664 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 00:06:35.695726 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:35.695685 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 00:06:35.695726 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:35.695696 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jqrtz for pod openshift-network-diagnostics/network-check-target-s2ksf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:35.695893 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:35.695753 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz podName:65a77142-1094-493e-862a-3a0a9fafe475 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:39.69573497 +0000 UTC m=+9.218702713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqrtz" (UniqueName: "kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz") pod "network-check-target-s2ksf" (UID: "65a77142-1094-493e-862a-3a0a9fafe475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:36.043152 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:36.042715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:36.043152 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:36.042856 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:36.043401 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:36.043306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:36.043458 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:36.043398 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:36.043517 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:36.043463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:36.043569 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:36.043538 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:38.043508 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:38.042971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:38.043508 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:38.043008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:38.043508 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:38.043008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:38.043508 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:38.043160 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:38.043508 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:38.043285 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:38.043508 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:38.043374 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:39.528831 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:39.528602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:39.528831 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:39.528788 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:39.529389 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:39.528853 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:06:47.528833107 +0000 UTC m=+17.051800834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:39.629952 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:39.629909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:39.630146 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:39.630066 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:39.630212 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:39.630153 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret podName:e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:47.630131863 +0000 UTC m=+17.153099600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret") pod "global-pull-secret-syncer-gjqf5" (UID: "e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9") : object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:39.730693 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:39.730652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:39.730875 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:39.730786 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 00:06:39.730875 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:39.730805 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 00:06:39.730875 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:39.730815 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jqrtz for pod openshift-network-diagnostics/network-check-target-s2ksf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:39.730875 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:39.730871 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz podName:65a77142-1094-493e-862a-3a0a9fafe475 nodeName:}" failed. No retries permitted until 2026-07-31 00:06:47.730854025 +0000 UTC m=+17.253821769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqrtz" (UniqueName: "kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz") pod "network-check-target-s2ksf" (UID: "65a77142-1094-493e-862a-3a0a9fafe475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:40.043113 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:40.043079 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:40.043317 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:40.043220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:40.043317 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:40.043230 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:40.043446 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:40.043324 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:40.043446 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:40.043227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:40.043446 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:40.043435 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:42.043512 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:42.043048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:42.043512 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:42.043217 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:42.043512 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:42.043300 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:42.043512 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:42.043368 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:42.043512 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:42.043419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:42.043512 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:42.043479 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:44.042580 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:44.042542 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:44.042904 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:44.042542 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:44.042904 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:44.042661 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:44.042904 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:44.042560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:44.042904 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:44.042722 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:44.042904 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:44.042813 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:46.043072 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:46.043043 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:46.043563 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:46.043167 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:46.043563 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:46.043180 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:46.043563 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:46.043207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:46.043563 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:46.043284 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:46.043563 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:46.043369 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:47.597019 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:47.596979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:47.597478 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:47.597111 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:47.597478 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:47.597187 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:07:03.597170194 +0000 UTC m=+33.120137917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:06:47.697732 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:47.697678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:47.697924 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:47.697812 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:47.697924 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:47.697887 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret podName:e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:03.697870475 +0000 UTC m=+33.220838221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret") pod "global-pull-secret-syncer-gjqf5" (UID: "e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9") : object "kube-system"/"original-pull-secret" not registered Jul 31 00:06:47.798857 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:47.798818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:47.799038 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:47.799016 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 00:06:47.799081 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:47.799044 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 00:06:47.799081 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:47.799053 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jqrtz for pod openshift-network-diagnostics/network-check-target-s2ksf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:47.799171 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:47.799129 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz podName:65a77142-1094-493e-862a-3a0a9fafe475 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:03.799099257 +0000 UTC m=+33.322067002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqrtz" (UniqueName: "kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz") pod "network-check-target-s2ksf" (UID: "65a77142-1094-493e-862a-3a0a9fafe475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:06:48.042671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:48.042582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:48.042671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:48.042603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:48.042671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:48.042603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:48.042917 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:48.042711 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:48.042917 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:48.042808 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:48.042998 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:48.042923 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:50.043348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:50.043310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:50.043770 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:50.043310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:50.043770 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:50.043440 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:50.043770 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:50.043537 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:50.043770 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:50.043310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:50.043770 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:50.043626 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:52.042748 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:52.042711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:52.042748 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:52.042739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:52.043229 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:52.042827 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:52.043229 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:52.042835 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:52.043229 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:52.042877 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:52.043229 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:52.042978 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:53.114063 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:53.113825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vqx58" event={"ID":"028bd031-00ad-4234-bb3d-fef11b3ff915","Type":"ContainerStarted","Data":"895aeb1194346cd4464322ed5d8ea1b6cc19a7a3efb1809ba0db2f638e667a72"} Jul 31 00:06:53.115823 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:53.115791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"70d0f18682d27a1d512912173507df0135151c167c4423057e8dc7f6803018ed"} Jul 31 00:06:53.117497 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:53.117468 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wt2nl" event={"ID":"fccb5b35-15ad-426c-a0c5-83eb68df8166","Type":"ContainerStarted","Data":"9aa38c0265a138f529229b9f434e63454223f0aedfd143b4f0783edfddaf1685"} Jul 31 00:06:53.118942 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:53.118920 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" event={"ID":"3820b175871ebd792360ac3026b04d32","Type":"ContainerStarted","Data":"ac8383eaf0c6cb9f0d5043a66fb29d822110e7173e7cd36c43135dc4c89ea876"} Jul 31 00:06:53.129604 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:53.129540 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vqx58" podStartSLOduration=2.349065824 podStartE2EDuration="22.129519767s" totalImagesPullingTime="19.780453943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:53.129062647 +0000 UTC m=+22.652030392" watchObservedRunningTime="2026-07-31 00:06:53.129519767 +0000 UTC m=+22.652487515" Jul 31 00:06:53.151017 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:53.150929 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wt2nl" podStartSLOduration=2.263622737 podStartE2EDuration="22.150910727s" totalImagesPullingTime="19.88728799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:53.150639379 +0000 UTC m=+22.673607161" watchObservedRunningTime="2026-07-31 00:06:53.150910727 +0000 UTC m=+22.673878472" Jul 31 00:06:53.169499 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:53.169443 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-191.ec2.internal" podStartSLOduration=22.169421017 podStartE2EDuration="22.169421017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:53.168895258 +0000 UTC m=+22.691863013" watchObservedRunningTime="2026-07-31 00:06:53.169421017 +0000 UTC m=+22.692388763" Jul 31 00:06:54.042803 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.042765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:54.042803 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.042791 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:54.043163 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.042765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:54.043163 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:54.042898 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:54.043163 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:54.042945 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:54.043163 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:54.043017 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:54.123864 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.123822 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5f6lz" event={"ID":"814f97fa-4c99-483b-a086-c3da15df60a6","Type":"ContainerStarted","Data":"3c2441bd8e215280812902ccbc36f5cd3e60a1339099d59b1c2c5ee5c46810ef"} Jul 31 00:06:54.125757 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.125728 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ck7r" event={"ID":"232e2b98-3c4e-415e-b538-6e350a6df518","Type":"ContainerStarted","Data":"418bfc46788b7ebbfe2c22ad092dbc87d601c5a38de7d3fdf8a5edb9d1664727"} Jul 31 00:06:54.127429 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.127386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6cvnn" event={"ID":"afbe38cc-74f5-4daf-a158-120a7a583821","Type":"ContainerStarted","Data":"6b29e63885989e2421abbae4707ac0f879fbcf9a599d3eb172d101dc5c1a1fa7"} Jul 31 00:06:54.129164 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.129141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" event={"ID":"b493d74f-267f-4463-b67d-0df930eba2ee","Type":"ContainerStarted","Data":"661a968dbe27be5549addb2b802c4281d13d9f644e8db5bcb346b810893118c9"} Jul 31 00:06:54.130802 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.130776 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f877c84-ab31-4503-9fcd-bb0f621e4ef9" containerID="1ee5c4cde38e5e8d7b0294dd1c8c98334c624495b8a7ac7d4728571d67731c9f" exitCode=0 Jul 31 00:06:54.130921 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.130863 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtk76" event={"ID":"1f877c84-ab31-4503-9fcd-bb0f621e4ef9","Type":"ContainerDied","Data":"1ee5c4cde38e5e8d7b0294dd1c8c98334c624495b8a7ac7d4728571d67731c9f"} Jul 31 00:06:54.134419 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.134394 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"b1badda3c738a52cc570d0b67b62406d4aeedded5041801f40f43dd41467dc84"} Jul 31 00:06:54.134544 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.134482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"ad15ad760a630b7b4aa51ea5b5a330f1316cf2befa2356db6020ed50c8905199"} Jul 31 00:06:54.134604 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.134541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"39f1217b6e0b9ce1d026c8e1c996d8cdf10bbf81eb3ac34c898bedbdbfd3f871"} Jul 31 00:06:54.134604 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.134590 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"1db14192ff6821d9d0b3a5c8681ccce75fae90fcb13012039ae20ca3ba2613f5"} Jul 31 00:06:54.134707 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.134646 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"d46a5929b4141e53b7d252ba1435bb18d88508e9a0a0a78de6f44a1f2cf9ecee"} Jul 31 00:06:54.136207 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.136184 2575 generic.go:356] "Generic (PLEG): container finished" podID="637e762b63558847f949d39093da072e" containerID="3af7bd547d42d0b1c52bfb05a90bbbaf0fd4048711dd290ef79c320c23a4dbc7" exitCode=0 Jul 31 00:06:54.136311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.136268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" event={"ID":"637e762b63558847f949d39093da072e","Type":"ContainerDied","Data":"3af7bd547d42d0b1c52bfb05a90bbbaf0fd4048711dd290ef79c320c23a4dbc7"} Jul 31 00:06:54.138231 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.138157 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bsbtr" event={"ID":"e470d8d6-7673-43d1-a5d3-ccf49c41c6bb","Type":"ContainerStarted","Data":"f35d243e4ae4b9b2678737105497193d09495c0a83f2650f6f97ada7c35e00e0"} Jul 31 00:06:54.150377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.150324 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9ck7r" podStartSLOduration=3.772347045 podStartE2EDuration="23.15030847s" totalImagesPullingTime="19.377961425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:54.15016496 +0000 UTC m=+23.673132715" watchObservedRunningTime="2026-07-31 00:06:54.15030847 +0000 UTC m=+23.673276214" Jul 31 00:06:54.150750 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.150725 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5f6lz" podStartSLOduration=3.555306038 podStartE2EDuration="23.150715303s" totalImagesPullingTime="19.595409265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:54.137821885 +0000 UTC m=+23.660789669" watchObservedRunningTime="2026-07-31 00:06:54.150715303 +0000 UTC m=+23.673683047" Jul 31 00:06:54.183972 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.183712 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6cvnn" podStartSLOduration=3.48414033 podStartE2EDuration="23.18368927s" totalImagesPullingTime="19.69954894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:54.183485991 +0000 UTC m=+23.706453738" watchObservedRunningTime="2026-07-31 00:06:54.18368927 +0000 UTC m=+23.706657016" Jul 31 00:06:54.208655 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.208609 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bsbtr" podStartSLOduration=3.757361055 podStartE2EDuration="23.208593952s" totalImagesPullingTime="19.451232897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:54.208026901 +0000 UTC m=+23.730994648" watchObservedRunningTime="2026-07-31 00:06:54.208593952 +0000 UTC m=+23.731561723" Jul 31 00:06:54.345968 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.345836 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:54.396261 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.396227 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:54.673641 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.673614 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 00:06:54.929393 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.929194 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T00:06:54.673633824Z","UUID":"8b69f000-df5c-4b33-a2e0-390b53ad9efc","Handler":null,"Name":"","Endpoint":""} Jul 31 00:06:54.930939 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.930915 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 00:06:54.931068 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:54.930962 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 00:06:55.142657 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:55.142616 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" event={"ID":"b493d74f-267f-4463-b67d-0df930eba2ee","Type":"ContainerStarted","Data":"6892009c127a40938fdee82fe87036d29646d8d5e5bc4321c7385d45a6e94ded"} Jul 31 00:06:55.144541 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:55.144504 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" event={"ID":"637e762b63558847f949d39093da072e","Type":"ContainerStarted","Data":"147116b76c7ad3d575d12b8af4754f16bc50556bff89773da0a1d58317d10237"} Jul 31 00:06:55.159563 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:55.159513 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-191.ec2.internal" podStartSLOduration=24.159496154 podStartE2EDuration="24.159496154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:55.159280386 +0000 UTC m=+24.682248134" watchObservedRunningTime="2026-07-31 00:06:55.159496154 +0000 UTC m=+24.682463899" Jul 31 00:06:56.042352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:56.042316 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:56.042563 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:56.042417 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:56.042563 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:56.042428 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:56.042563 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:56.042454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:56.042563 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:56.042528 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:56.042721 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:56.042593 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:56.149867 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:56.149833 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" event={"ID":"b493d74f-267f-4463-b67d-0df930eba2ee","Type":"ContainerStarted","Data":"ad703ea17fe9b9d8b976c48c7785fabc77e9de042d2d67206e49f104658cd463"} Jul 31 00:06:56.153543 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:56.153514 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"5e072b2026bc2cd2f5c2ed522f36cd4587f227adf1f0109811d4893ea1736a25"} Jul 31 00:06:56.167195 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:56.167145 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4dbp" podStartSLOduration=3.195323303 podStartE2EDuration="25.167111463s" totalImagesPullingTime="21.97178816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:56.166555737 +0000 UTC m=+25.689523503" watchObservedRunningTime="2026-07-31 00:06:56.167111463 +0000 UTC m=+25.690079207" Jul 31 00:06:56.842750 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:56.842707 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:56.843387 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:56.843362 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:57.156221 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:57.156144 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:57.156638 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:57.156621 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6cvnn" Jul 31 00:06:57.208019 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:57.207985 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:58.042998 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:58.042952 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:06:58.043186 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:58.042962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:06:58.043186 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:58.043066 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:06:58.043186 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:58.043090 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:06:58.043333 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:58.043203 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:06:58.043333 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:06:58.043292 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:06:58.075374 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:58.075267 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:58.175886 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:58.175850 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:58.278928 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:58.278885 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:58.326301 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:58.326213 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:59.130659 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.130481 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:59.162978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.162944 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f877c84-ab31-4503-9fcd-bb0f621e4ef9" containerID="169c3ed57e497de43ae0876a66b1eeb43e15dead7268d945f6d4166feca24ed6" exitCode=0 Jul 31 00:06:59.163190 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.163030 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtk76" event={"ID":"1f877c84-ab31-4503-9fcd-bb0f621e4ef9","Type":"ContainerDied","Data":"169c3ed57e497de43ae0876a66b1eeb43e15dead7268d945f6d4166feca24ed6"} Jul 31 00:06:59.166839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.166813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" event={"ID":"b0967a28-06e3-4d79-9e5e-930d0d9c2152","Type":"ContainerStarted","Data":"4647b32fba4c4ecd866153aed3797db37db7d5e0308c3b86a2dd536b93d02d64"} Jul 31 00:06:59.167096 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.167081 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:59.167189 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.167164 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:59.182634 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.182608 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:06:59.186621 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.186581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:06:59.206510 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:06:59.206437 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" podStartSLOduration=7.870751892 podStartE2EDuration="28.206419872s" totalImagesPullingTime="20.33566798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:06:59.205963281 +0000 UTC m=+28.728931025" watchObservedRunningTime="2026-07-31 00:06:59.206419872 +0000 UTC m=+28.729387617" Jul 31 00:07:00.042625 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:00.042586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:00.042625 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:00.042614 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:00.042857 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:00.042620 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:07:00.042857 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:00.042734 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:07:00.042986 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:00.042859 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:07:00.042986 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:00.042964 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:07:00.169891 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:00.169854 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:07:00.184709 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:00.184679 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:07:00.579397 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:00.579356 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:00.629221 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:00.629191 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:01.018973 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:01.018802 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s2ksf"] Jul 31 00:07:01.019222 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:01.019055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:01.019222 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:01.019182 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:07:01.019423 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:01.019403 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjqf5"] Jul 31 00:07:01.019493 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:01.019483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:01.019583 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:01.019564 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:07:01.021271 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:01.021244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xpcj"] Jul 31 00:07:01.021404 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:01.021350 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:07:01.021507 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:01.021462 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:07:01.173944 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:01.173839 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f877c84-ab31-4503-9fcd-bb0f621e4ef9" containerID="8e16c262243a653ee630d8d961a8b4ec94cad0e26cdc85593219a052cd12f381" exitCode=0 Jul 31 00:07:01.173944 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:01.173926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtk76" event={"ID":"1f877c84-ab31-4503-9fcd-bb0f621e4ef9","Type":"ContainerDied","Data":"8e16c262243a653ee630d8d961a8b4ec94cad0e26cdc85593219a052cd12f381"} Jul 31 00:07:03.042755 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.042719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:07:03.043168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.042766 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:03.043168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.042721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:03.043168 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.042870 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:07:03.043168 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.042934 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:07:03.043168 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.042986 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:07:03.095999 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.095961 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:03.180878 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.180790 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f877c84-ab31-4503-9fcd-bb0f621e4ef9" containerID="903b6d85c4ad9622c699fcefaede1c6cca2b1fa819b20b288615cbad6b1a16e5" exitCode=0 Jul 31 00:07:03.180878 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.180841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtk76" event={"ID":"1f877c84-ab31-4503-9fcd-bb0f621e4ef9","Type":"ContainerDied","Data":"903b6d85c4ad9622c699fcefaede1c6cca2b1fa819b20b288615cbad6b1a16e5"} Jul 31 00:07:03.611016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.610922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:07:03.611207 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.611099 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:07:03.611207 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.611198 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:07:35.611178556 +0000 UTC m=+65.134146297 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 00:07:03.711322 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.711291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:03.711491 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.711475 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 00:07:03.711569 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.711547 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret podName:e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:35.711526716 +0000 UTC m=+65.234494456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret") pod "global-pull-secret-syncer-gjqf5" (UID: "e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9") : object "kube-system"/"original-pull-secret" not registered Jul 31 00:07:03.811813 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:03.811772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:03.811994 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.811970 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 00:07:03.812076 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.812005 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 00:07:03.812076 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.812019 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jqrtz for pod openshift-network-diagnostics/network-check-target-s2ksf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:07:03.812213 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:03.812087 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz podName:65a77142-1094-493e-862a-3a0a9fafe475 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:35.812065182 +0000 UTC m=+65.335032921 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqrtz" (UniqueName: "kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz") pod "network-check-target-s2ksf" (UID: "65a77142-1094-493e-862a-3a0a9fafe475") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 00:07:05.042542 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.042500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:05.042960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.042500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:07:05.043333 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:05.043271 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjqf5" podUID="e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9" Jul 31 00:07:05.043420 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.043340 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:05.043489 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:05.043471 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:07:05.043592 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:05.043571 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2ksf" podUID="65a77142-1094-493e-862a-3a0a9fafe475" Jul 31 00:07:05.759430 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.759362 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-191.ec2.internal" event="NodeReady" Jul 31 00:07:05.759582 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.759499 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 00:07:05.799301 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.799269 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gtc62"] Jul 31 00:07:05.803418 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.803395 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-87299"] Jul 31 00:07:05.803550 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.803525 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:05.806011 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.805983 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 00:07:05.806011 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.806009 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nnjn4\"" Jul 31 00:07:05.806216 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.805992 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 00:07:05.807361 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.807341 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87299" Jul 31 00:07:05.809588 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.809566 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 00:07:05.809727 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.809708 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 00:07:05.809797 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.809783 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 00:07:05.809855 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.809840 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5mc7x\"" Jul 31 00:07:05.812181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.811857 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gtc62"] Jul 31 00:07:05.814502 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.814471 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87299"] Jul 31 00:07:05.873536 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.873497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:05.927260 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.927227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63977669-cfbc-4363-9091-f32b6585ae23-cert\") pod \"ingress-canary-87299\" (UID: \"63977669-cfbc-4363-9091-f32b6585ae23\") " pod="openshift-ingress-canary/ingress-canary-87299" Jul 31 00:07:05.927455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.927267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa00574-87fc-449d-a891-4b43e2f0d014-config-volume\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:05.927455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.927308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ssk\" (UniqueName: \"kubernetes.io/projected/7aa00574-87fc-449d-a891-4b43e2f0d014-kube-api-access-69ssk\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:05.927455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.927354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa00574-87fc-449d-a891-4b43e2f0d014-tmp-dir\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:05.927455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.927421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtpb\" (UniqueName: \"kubernetes.io/projected/63977669-cfbc-4363-9091-f32b6585ae23-kube-api-access-2dtpb\") pod \"ingress-canary-87299\" (UID: \"63977669-cfbc-4363-9091-f32b6585ae23\") " pod="openshift-ingress-canary/ingress-canary-87299" Jul 31 00:07:05.927455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.927451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:05.974310 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:05.974271 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:06.028226 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.028131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa00574-87fc-449d-a891-4b43e2f0d014-tmp-dir\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:06.028226 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.028218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtpb\" (UniqueName: \"kubernetes.io/projected/63977669-cfbc-4363-9091-f32b6585ae23-kube-api-access-2dtpb\") pod \"ingress-canary-87299\" (UID: \"63977669-cfbc-4363-9091-f32b6585ae23\") " pod="openshift-ingress-canary/ingress-canary-87299" Jul 31 00:07:06.028431 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.028245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:06.028431 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.028292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63977669-cfbc-4363-9091-f32b6585ae23-cert\") pod \"ingress-canary-87299\" (UID: \"63977669-cfbc-4363-9091-f32b6585ae23\") " pod="openshift-ingress-canary/ingress-canary-87299" Jul 31 00:07:06.028431 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.028318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa00574-87fc-449d-a891-4b43e2f0d014-config-volume\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:06.028431 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.028358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69ssk\" (UniqueName: \"kubernetes.io/projected/7aa00574-87fc-449d-a891-4b43e2f0d014-kube-api-access-69ssk\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:06.028431 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:06.028405 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:07:06.028652 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:06.028475 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:06.52845845 +0000 UTC m=+36.051426173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:07:06.028834 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.028809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa00574-87fc-449d-a891-4b43e2f0d014-tmp-dir\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:06.029238 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.029047 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa00574-87fc-449d-a891-4b43e2f0d014-config-volume\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:06.033061 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.033038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63977669-cfbc-4363-9091-f32b6585ae23-cert\") pod \"ingress-canary-87299\" (UID: \"63977669-cfbc-4363-9091-f32b6585ae23\") " pod="openshift-ingress-canary/ingress-canary-87299" Jul 31 00:07:06.037226 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.037167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ssk\" (UniqueName: \"kubernetes.io/projected/7aa00574-87fc-449d-a891-4b43e2f0d014-kube-api-access-69ssk\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:06.037415 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.037393 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtpb\" (UniqueName: \"kubernetes.io/projected/63977669-cfbc-4363-9091-f32b6585ae23-kube-api-access-2dtpb\") pod \"ingress-canary-87299\" (UID: \"63977669-cfbc-4363-9091-f32b6585ae23\") " pod="openshift-ingress-canary/ingress-canary-87299" Jul 31 00:07:06.122526 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.122484 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87299" Jul 31 00:07:06.531582 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:06.531545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:06.531897 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:06.531728 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:07:06.531897 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:06.531812 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:07.531788355 +0000 UTC m=+37.054756102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:07:07.042905 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.042865 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:07:07.043104 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.042865 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:07.043104 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.043021 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:07.045674 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.045639 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 00:07:07.045821 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.045753 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 00:07:07.046013 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.045998 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 00:07:07.047631 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.047295 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 00:07:07.047631 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.047435 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tbfb2\"" Jul 31 00:07:07.047631 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.047504 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2dmpl\"" Jul 31 00:07:07.416206 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.416178 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87299"] Jul 31 00:07:07.419717 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:07:07.419686 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63977669_cfbc_4363_9091_f32b6585ae23.slice/crio-c002262b0788d43fc4881bc02fa96d2b1b7ba3802402ff4583eeaecf1f62b9ad WatchSource:0}: Error finding container c002262b0788d43fc4881bc02fa96d2b1b7ba3802402ff4583eeaecf1f62b9ad: Status 404 returned error can't find the container with id c002262b0788d43fc4881bc02fa96d2b1b7ba3802402ff4583eeaecf1f62b9ad Jul 31 00:07:07.538605 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:07.538428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:07.538698 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:07.538578 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:07:07.538734 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:07.538708 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:09.538691899 +0000 UTC m=+39.061659635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:07:08.198091 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:08.198000 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f877c84-ab31-4503-9fcd-bb0f621e4ef9" containerID="242d828ca2d4748ab395b82594bb05ad5cb272e8d77507163994ba72dcff01ad" exitCode=0 Jul 31 00:07:08.198271 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:08.198135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtk76" event={"ID":"1f877c84-ab31-4503-9fcd-bb0f621e4ef9","Type":"ContainerDied","Data":"242d828ca2d4748ab395b82594bb05ad5cb272e8d77507163994ba72dcff01ad"} Jul 31 00:07:08.199521 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:08.199483 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87299" event={"ID":"63977669-cfbc-4363-9091-f32b6585ae23","Type":"ContainerStarted","Data":"c002262b0788d43fc4881bc02fa96d2b1b7ba3802402ff4583eeaecf1f62b9ad"} Jul 31 00:07:09.205999 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:09.205965 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f877c84-ab31-4503-9fcd-bb0f621e4ef9" containerID="a83c6342970c539d719a076457b3852635b8ae2d741a71fcb9fc161e39ea43ab" exitCode=0 Jul 31 00:07:09.206471 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:09.206018 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtk76" event={"ID":"1f877c84-ab31-4503-9fcd-bb0f621e4ef9","Type":"ContainerDied","Data":"a83c6342970c539d719a076457b3852635b8ae2d741a71fcb9fc161e39ea43ab"} Jul 31 00:07:09.555533 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:09.555407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:09.555748 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:09.555584 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:07:09.555748 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:09.555657 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:13.555640397 +0000 UTC m=+43.078608120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:07:09.960986 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:09.960953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:10.063248 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.063203 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:10.170070 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.170029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:10.212071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.212031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtk76" event={"ID":"1f877c84-ab31-4503-9fcd-bb0f621e4ef9","Type":"ContainerStarted","Data":"32005ba5f4a32af7b2c9d307fd3edd66c02f3f3f4be33b2de2d799d190b7a144"} Jul 31 00:07:10.213474 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.213448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87299" event={"ID":"63977669-cfbc-4363-9091-f32b6585ae23","Type":"ContainerStarted","Data":"e49d2b1b552f84149c7b77e48e2c73a8129d24a1f6bd5b1092bdb547d53881fb"} Jul 31 00:07:10.232354 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.232307 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wtk76" podStartSLOduration=8.481257544 podStartE2EDuration="39.232291249s" totalImagesPullingTime="30.751033705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:07:10.230880669 +0000 UTC m=+39.753848435" watchObservedRunningTime="2026-07-31 00:07:10.232291249 +0000 UTC m=+39.755258994" Jul 31 00:07:10.240755 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.240725 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:10.248584 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.248531 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-87299" podStartSLOduration=2.676031993 podStartE2EDuration="5.248508183s" totalImagesPullingTime="2.57247619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:07:10.247436773 +0000 UTC m=+39.770404517" watchObservedRunningTime="2026-07-31 00:07:10.248508183 +0000 UTC m=+39.771475929" Jul 31 00:07:10.320003 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.319967 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:10.371871 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:10.371787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:13.588686 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:13.588644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:13.589070 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:13.588812 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:07:13.589070 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:13.588879 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:21.588861672 +0000 UTC m=+51.111829395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:07:14.168160 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:14.168088 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:14.268880 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:14.268844 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:14.331715 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:14.331681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:21.644904 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:21.644864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:21.645390 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:21.645049 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:07:21.645390 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:21.645160 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:07:37.645136647 +0000 UTC m=+67.168104384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:07:32.206501 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:32.206471 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gflzn" Jul 31 00:07:32.353298 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:32.353242 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:32.404605 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:32.404569 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:35.154976 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.154936 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:35.173469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.173435 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr"] Jul 31 00:07:35.177047 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.177026 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:07:35.179933 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.179904 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 31 00:07:35.181313 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.181290 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 31 00:07:35.181443 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.181301 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 31 00:07:35.181606 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.181590 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2592k\"" Jul 31 00:07:35.181680 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.181665 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 31 00:07:35.187348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.187329 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr"] Jul 31 00:07:35.194610 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.194591 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl"] Jul 31 00:07:35.196319 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.196301 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.198908 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.198891 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 31 00:07:35.199020 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.198922 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 31 00:07:35.199020 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.198936 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 31 00:07:35.199020 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.198925 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 31 00:07:35.209288 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.209266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl"] Jul 31 00:07:35.230094 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.230059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:35.345635 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.345603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26d45\" (UniqueName: \"kubernetes.io/projected/6b92aa78-3258-45a5-b53b-2813bd1b1586-kube-api-access-26d45\") pod \"managed-serviceaccount-addon-agent-869bb84c64-tg7lr\" (UID: \"6b92aa78-3258-45a5-b53b-2813bd1b1586\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:07:35.345635 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.345640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.345848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.345662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.345848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.345751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-ca\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.345848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.345775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92bnv\" (UniqueName: \"kubernetes.io/projected/7a288f42-eede-4280-aa3d-c91b26aee6a9-kube-api-access-92bnv\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.345848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.345799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b92aa78-3258-45a5-b53b-2813bd1b1586-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869bb84c64-tg7lr\" (UID: \"6b92aa78-3258-45a5-b53b-2813bd1b1586\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:07:35.345848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.345816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a288f42-eede-4280-aa3d-c91b26aee6a9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.345998 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.345856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-hub\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.446985 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.446913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.446985 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.446955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.447071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.447010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-ca\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.447071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.447033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bnv\" (UniqueName: \"kubernetes.io/projected/7a288f42-eede-4280-aa3d-c91b26aee6a9-kube-api-access-92bnv\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.447071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.447060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b92aa78-3258-45a5-b53b-2813bd1b1586-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869bb84c64-tg7lr\" (UID: \"6b92aa78-3258-45a5-b53b-2813bd1b1586\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:07:35.447243 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.447084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a288f42-eede-4280-aa3d-c91b26aee6a9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.447243 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.447165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-hub\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.447243 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.447217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26d45\" (UniqueName: \"kubernetes.io/projected/6b92aa78-3258-45a5-b53b-2813bd1b1586-kube-api-access-26d45\") pod \"managed-serviceaccount-addon-agent-869bb84c64-tg7lr\" (UID: \"6b92aa78-3258-45a5-b53b-2813bd1b1586\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:07:35.447944 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.447915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a288f42-eede-4280-aa3d-c91b26aee6a9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.450534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.450508 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.450704 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.450687 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.450745 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.450697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b92aa78-3258-45a5-b53b-2813bd1b1586-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869bb84c64-tg7lr\" (UID: \"6b92aa78-3258-45a5-b53b-2813bd1b1586\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:07:35.450745 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.450705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-ca\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.450745 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.450697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a288f42-eede-4280-aa3d-c91b26aee6a9-hub\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.455046 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.455014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26d45\" (UniqueName: \"kubernetes.io/projected/6b92aa78-3258-45a5-b53b-2813bd1b1586-kube-api-access-26d45\") pod \"managed-serviceaccount-addon-agent-869bb84c64-tg7lr\" (UID: \"6b92aa78-3258-45a5-b53b-2813bd1b1586\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:07:35.455201 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.455014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92bnv\" (UniqueName: \"kubernetes.io/projected/7a288f42-eede-4280-aa3d-c91b26aee6a9-kube-api-access-92bnv\") pod \"cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl\" (UID: \"7a288f42-eede-4280-aa3d-c91b26aee6a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.496428 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.496391 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:07:35.504097 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.504061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:07:35.630592 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.630558 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr"] Jul 31 00:07:35.633377 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:07:35.633344 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b92aa78_3258_45a5_b53b_2813bd1b1586.slice/crio-c6690b46d0acac8d31195a77fdda0b1e6f552cf21abcb8d1fd27c07892c2916f WatchSource:0}: Error finding container c6690b46d0acac8d31195a77fdda0b1e6f552cf21abcb8d1fd27c07892c2916f: Status 404 returned error can't find the container with id c6690b46d0acac8d31195a77fdda0b1e6f552cf21abcb8d1fd27c07892c2916f Jul 31 00:07:35.645408 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.645347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl"] Jul 31 00:07:35.647901 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:07:35.647876 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a288f42_eede_4280_aa3d_c91b26aee6a9.slice/crio-4d2a9ca5a7fcb27ea39bd379443c81d2db697fddeddc819f23fc8f26164faeb8 WatchSource:0}: Error finding container 4d2a9ca5a7fcb27ea39bd379443c81d2db697fddeddc819f23fc8f26164faeb8: Status 404 returned error can't find the container with id 4d2a9ca5a7fcb27ea39bd379443c81d2db697fddeddc819f23fc8f26164faeb8 Jul 31 00:07:35.648056 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.648039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:07:35.650412 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.650389 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 00:07:35.658832 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:35.658808 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 00:07:35.658943 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:35.658874 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:08:39.658853009 +0000 UTC m=+129.181820732 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : secret "metrics-daemon-secret" not found Jul 31 00:07:35.749466 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.749380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:35.751956 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.751938 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 00:07:35.762383 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.762358 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9-original-pull-secret\") pod \"global-pull-secret-syncer-gjqf5\" (UID: \"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9\") " pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:35.849909 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.849849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:35.852428 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.852410 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 00:07:35.862603 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.862582 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 00:07:35.867588 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.867571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjqf5" Jul 31 00:07:35.873562 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.873533 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrtz\" (UniqueName: \"kubernetes.io/projected/65a77142-1094-493e-862a-3a0a9fafe475-kube-api-access-jqrtz\") pod \"network-check-target-s2ksf\" (UID: \"65a77142-1094-493e-862a-3a0a9fafe475\") " pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:35.986714 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:35.986690 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjqf5"] Jul 31 00:07:35.989029 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:07:35.989004 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f6ffa3_e0a4_49f0_bfe6_7ca019be77b9.slice/crio-8073600b06292084dd148e054f7a11634cc965f1c11797a499d11d95dbda75fb WatchSource:0}: Error finding container 8073600b06292084dd148e054f7a11634cc965f1c11797a499d11d95dbda75fb: Status 404 returned error can't find the container with id 8073600b06292084dd148e054f7a11634cc965f1c11797a499d11d95dbda75fb Jul 31 00:07:36.164884 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:36.164855 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2dmpl\"" Jul 31 00:07:36.172895 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:36.172868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:36.273100 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:36.273063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjqf5" event={"ID":"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9","Type":"ContainerStarted","Data":"8073600b06292084dd148e054f7a11634cc965f1c11797a499d11d95dbda75fb"} Jul 31 00:07:36.274285 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:36.274237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" event={"ID":"6b92aa78-3258-45a5-b53b-2813bd1b1586","Type":"ContainerStarted","Data":"c6690b46d0acac8d31195a77fdda0b1e6f552cf21abcb8d1fd27c07892c2916f"} Jul 31 00:07:36.275459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:36.275432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" event={"ID":"7a288f42-eede-4280-aa3d-c91b26aee6a9","Type":"ContainerStarted","Data":"4d2a9ca5a7fcb27ea39bd379443c81d2db697fddeddc819f23fc8f26164faeb8"} Jul 31 00:07:36.312757 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:36.312725 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s2ksf"] Jul 31 00:07:36.316947 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:07:36.316924 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a77142_1094_493e_862a_3a0a9fafe475.slice/crio-e51b8c492259c7e4ad4baa9ceab93a54e370cdcdbe4bde1aebd705ab42cb178b WatchSource:0}: Error finding container e51b8c492259c7e4ad4baa9ceab93a54e370cdcdbe4bde1aebd705ab42cb178b: Status 404 returned error can't find the container with id e51b8c492259c7e4ad4baa9ceab93a54e370cdcdbe4bde1aebd705ab42cb178b Jul 31 00:07:37.281186 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:37.281147 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s2ksf" event={"ID":"65a77142-1094-493e-862a-3a0a9fafe475","Type":"ContainerStarted","Data":"e51b8c492259c7e4ad4baa9ceab93a54e370cdcdbe4bde1aebd705ab42cb178b"} Jul 31 00:07:37.666168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:37.666129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:07:37.666411 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:37.666297 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:07:37.666411 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:07:37.666361 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:08:09.666341933 +0000 UTC m=+99.189309661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:07:37.840754 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:37.840707 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:07:44.303589 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.303550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjqf5" event={"ID":"e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9","Type":"ContainerStarted","Data":"eac673c711bd83ee1e71368f434ad556fb32ee5473780373785b68f9b4e5ba31"} Jul 31 00:07:44.305109 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.305084 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s2ksf" event={"ID":"65a77142-1094-493e-862a-3a0a9fafe475","Type":"ContainerStarted","Data":"5fe7455e3de01e352c44ed593b6f13d9eb4b8ed7c8edde209bb8162e041f3661"} Jul 31 00:07:44.305257 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.305156 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:07:44.306436 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.306404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" event={"ID":"6b92aa78-3258-45a5-b53b-2813bd1b1586","Type":"ContainerStarted","Data":"5ba4adcde73c5a0a6c6502715110fe4bb106b133f0f52afa6108a6bf4f50812c"} Jul 31 00:07:44.308343 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.308320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" event={"ID":"7a288f42-eede-4280-aa3d-c91b26aee6a9","Type":"ContainerStarted","Data":"702414f6ba60d8e44733d2335c11764d444eaea72d3a68be5c2a634d68e4d8f5"} Jul 31 00:07:44.308462 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.308356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" event={"ID":"7a288f42-eede-4280-aa3d-c91b26aee6a9","Type":"ContainerStarted","Data":"d9a871f3c4507102f6357e9ab5ce118a7c96154c964cfb2a5b4db5e89e3ce84d"} Jul 31 00:07:44.308462 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.308374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" event={"ID":"7a288f42-eede-4280-aa3d-c91b26aee6a9","Type":"ContainerStarted","Data":"59796d8e00add4758875a88633c60ffcc73c473330488dbdd29d0e3b30f5d74e"} Jul 31 00:07:44.317849 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.317807 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gjqf5" podStartSLOduration=65.627367547 podStartE2EDuration="1m13.317791807s" totalImagesPullingTime="7.69042426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:07:44.317080018 +0000 UTC m=+73.840047763" watchObservedRunningTime="2026-07-31 00:07:44.317791807 +0000 UTC m=+73.840759530" Jul 31 00:07:44.335597 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.335521 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" podStartSLOduration=1.299156365 podStartE2EDuration="9.335497058s" totalImagesPullingTime="8.036340693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:07:44.333773962 +0000 UTC m=+73.856741707" watchObservedRunningTime="2026-07-31 00:07:44.335497058 +0000 UTC m=+73.858464804" Jul 31 00:07:44.347605 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.347553 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s2ksf" podStartSLOduration=65.998506641 podStartE2EDuration="1m13.347537182s" totalImagesPullingTime="7.349030541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:07:44.347213364 +0000 UTC m=+73.870181109" watchObservedRunningTime="2026-07-31 00:07:44.347537182 +0000 UTC m=+73.870504928" Jul 31 00:07:44.360946 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:07:44.360890 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" podStartSLOduration=1.32315948 podStartE2EDuration="9.360870674s" totalImagesPullingTime="8.037711194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:07:44.36056967 +0000 UTC m=+73.883537415" watchObservedRunningTime="2026-07-31 00:07:44.360870674 +0000 UTC m=+73.883838419" Jul 31 00:08:02.431096 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:08:02.430974 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:08:02.532002 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:08:02.531969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:08:09.696386 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:08:09.696341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:08:09.696845 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:08:09.696484 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:08:09.696845 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:08:09.696551 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:09:13.696531 +0000 UTC m=+163.219498744 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:08:15.313922 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:08:15.313891 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s2ksf" Jul 31 00:08:39.703360 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:08:39.703318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:08:39.703856 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:08:39.703454 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 00:08:39.703856 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:08:39.703523 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs podName:7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d nodeName:}" failed. No retries permitted until 2026-07-31 00:10:41.703506318 +0000 UTC m=+251.226474042 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs") pod "network-metrics-daemon-6xpcj" (UID: "7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d") : secret "metrics-daemon-secret" not found Jul 31 00:09:08.816446 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:09:08.816400 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gtc62" podUID="7aa00574-87fc-449d-a891-4b43e2f0d014" Jul 31 00:09:09.558102 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:09.558069 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtc62" Jul 31 00:09:10.055697 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:09:10.055644 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6xpcj" podUID="7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d" Jul 31 00:09:13.741933 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:13.741871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:09:13.742353 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:09:13.742039 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 00:09:13.742353 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:09:13.742133 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls podName:7aa00574-87fc-449d-a891-4b43e2f0d014 nodeName:}" failed. No retries permitted until 2026-07-31 00:11:15.742099774 +0000 UTC m=+285.265067497 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls") pod "dns-default-gtc62" (UID: "7aa00574-87fc-449d-a891-4b43e2f0d014") : secret "dns-default-metrics-tls" not found Jul 31 00:09:23.042659 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:23.042563 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:09:44.659291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:44.659252 2575 generic.go:356] "Generic (PLEG): container finished" podID="6b92aa78-3258-45a5-b53b-2813bd1b1586" containerID="5ba4adcde73c5a0a6c6502715110fe4bb106b133f0f52afa6108a6bf4f50812c" exitCode=255 Jul 31 00:09:44.659668 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:44.659323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" event={"ID":"6b92aa78-3258-45a5-b53b-2813bd1b1586","Type":"ContainerDied","Data":"5ba4adcde73c5a0a6c6502715110fe4bb106b133f0f52afa6108a6bf4f50812c"} Jul 31 00:09:44.659668 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:44.659650 2575 scope.go:117] "RemoveContainer" containerID="5ba4adcde73c5a0a6c6502715110fe4bb106b133f0f52afa6108a6bf4f50812c" Jul 31 00:09:45.497581 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:45.497534 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" Jul 31 00:09:45.663348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:45.663309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869bb84c64-tg7lr" event={"ID":"6b92aa78-3258-45a5-b53b-2813bd1b1586","Type":"ContainerStarted","Data":"fbd9aad105fd95a1966db3755af4d82adb07002e9b85443db318eb5f2f46542a"} Jul 31 00:09:51.872214 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:09:51.872177 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:14.867191 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:14.867156 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:14.968047 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:14.968012 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:17.072573 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:17.072534 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:17.122833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:17.122807 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:18.320369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.320334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:18.419475 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.419437 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:18.519394 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.519361 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:18.624200 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.624099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:18.647050 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.647015 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r74xz"] Jul 31 00:10:18.649040 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.649023 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.651370 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.651344 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 00:10:18.652897 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.652868 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 00:10:18.653028 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.652925 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f672z\"" Jul 31 00:10:18.653028 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.652940 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 00:10:18.653943 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.653925 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 00:10:18.662297 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.662270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r74xz"] Jul 31 00:10:18.680694 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.680652 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:18.796414 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.796373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7htp\" (UniqueName: \"kubernetes.io/projected/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-api-access-j7htp\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.796414 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.796423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31f3ae79-4f9e-46cd-bd66-25e2179d6915-crio-socket\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.796638 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.796441 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31f3ae79-4f9e-46cd-bd66-25e2179d6915-data-volume\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.796638 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.796488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.796638 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.796530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.897220 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.897144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.897220 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.897196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.897379 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.897238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7htp\" (UniqueName: \"kubernetes.io/projected/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-api-access-j7htp\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.897379 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:18.897256 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 00:10:18.897379 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.897265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31f3ae79-4f9e-46cd-bd66-25e2179d6915-crio-socket\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.897379 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.897281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31f3ae79-4f9e-46cd-bd66-25e2179d6915-data-volume\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.897379 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:18.897330 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm podName:31f3ae79-4f9e-46cd-bd66-25e2179d6915 nodeName:}" failed. No retries permitted until 2026-07-31 00:10:19.397310045 +0000 UTC m=+228.920277768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r74xz" (UID: "31f3ae79-4f9e-46cd-bd66-25e2179d6915") : configmap "kube-rbac-proxy" not found Jul 31 00:10:18.897601 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:18.897381 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 00:10:18.897601 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.897382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31f3ae79-4f9e-46cd-bd66-25e2179d6915-crio-socket\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.897601 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:18.897451 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls podName:31f3ae79-4f9e-46cd-bd66-25e2179d6915 nodeName:}" failed. No retries permitted until 2026-07-31 00:10:19.397433048 +0000 UTC m=+228.920400789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r74xz" (UID: "31f3ae79-4f9e-46cd-bd66-25e2179d6915") : secret "insights-runtime-extractor-tls" not found Jul 31 00:10:18.897601 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.897579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31f3ae79-4f9e-46cd-bd66-25e2179d6915-data-volume\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.905458 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.905425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7htp\" (UniqueName: \"kubernetes.io/projected/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-api-access-j7htp\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:18.936751 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:18.936719 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:19.211193 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:19.211157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:19.311244 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:19.311211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:19.400520 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:19.400480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:19.400914 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:19.400560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:19.400914 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:19.400632 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 00:10:19.400914 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:19.400635 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 00:10:19.400914 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:19.400687 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm podName:31f3ae79-4f9e-46cd-bd66-25e2179d6915 nodeName:}" failed. No retries permitted until 2026-07-31 00:10:20.400672142 +0000 UTC m=+229.923639870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r74xz" (UID: "31f3ae79-4f9e-46cd-bd66-25e2179d6915") : configmap "kube-rbac-proxy" not found Jul 31 00:10:19.400914 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:19.400700 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls podName:31f3ae79-4f9e-46cd-bd66-25e2179d6915 nodeName:}" failed. No retries permitted until 2026-07-31 00:10:20.40069447 +0000 UTC m=+229.923662196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r74xz" (UID: "31f3ae79-4f9e-46cd-bd66-25e2179d6915") : secret "insights-runtime-extractor-tls" not found Jul 31 00:10:20.409642 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.409588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:20.409642 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.409654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:20.410081 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:20.409714 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 00:10:20.410081 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:20.409758 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 00:10:20.410081 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:20.409782 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm podName:31f3ae79-4f9e-46cd-bd66-25e2179d6915 nodeName:}" failed. No retries permitted until 2026-07-31 00:10:22.409766454 +0000 UTC m=+231.932734181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r74xz" (UID: "31f3ae79-4f9e-46cd-bd66-25e2179d6915") : configmap "kube-rbac-proxy" not found Jul 31 00:10:20.410081 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:20.409798 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls podName:31f3ae79-4f9e-46cd-bd66-25e2179d6915 nodeName:}" failed. No retries permitted until 2026-07-31 00:10:22.409790587 +0000 UTC m=+231.932758313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r74xz" (UID: "31f3ae79-4f9e-46cd-bd66-25e2179d6915") : secret "insights-runtime-extractor-tls" not found Jul 31 00:10:20.610299 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.610260 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:20.711012 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.710901 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:20.808096 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.808057 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-dt9fm"] Jul 31 00:10:20.809935 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.809912 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:20.812378 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.812359 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 31 00:10:20.812482 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.812435 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 31 00:10:20.813571 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.813553 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 31 00:10:20.813674 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.813589 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qgclq\"" Jul 31 00:10:20.813674 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.813618 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 31 00:10:20.820662 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.820634 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-dt9fm"] Jul 31 00:10:20.847355 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.847321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:20.896256 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.896227 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:20.913371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.913336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-signing-cabundle\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:20.913511 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.913377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-signing-key\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:20.913511 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:20.913408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bf7\" (UniqueName: \"kubernetes.io/projected/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-kube-api-access-57bf7\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:21.014498 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.014398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-signing-cabundle\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:21.014498 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.014439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-signing-key\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:21.014498 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.014468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bf7\" (UniqueName: \"kubernetes.io/projected/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-kube-api-access-57bf7\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:21.015195 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.015170 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-signing-cabundle\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:21.016859 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.016838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-signing-key\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:21.022276 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.022251 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bf7\" (UniqueName: \"kubernetes.io/projected/f3d38f2b-c3d3-41d8-9129-7e6b6208655a-kube-api-access-57bf7\") pod \"service-ca-98c88fbb5-dt9fm\" (UID: \"f3d38f2b-c3d3-41d8-9129-7e6b6208655a\") " pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:21.119376 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.119330 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" Jul 31 00:10:21.214099 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.214074 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5f6lz_814f97fa-4c99-483b-a086-c3da15df60a6/dns-node-resolver/0.log" Jul 31 00:10:21.241873 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.241852 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-dt9fm"] Jul 31 00:10:21.244325 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:10:21.244290 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d38f2b_c3d3_41d8_9129_7e6b6208655a.slice/crio-730af99ace2652021d4d7abedb363872bd1dddf6d8555299173135e8628effa5 WatchSource:0}: Error finding container 730af99ace2652021d4d7abedb363872bd1dddf6d8555299173135e8628effa5: Status 404 returned error can't find the container with id 730af99ace2652021d4d7abedb363872bd1dddf6d8555299173135e8628effa5 Jul 31 00:10:21.694800 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.694766 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:21.766282 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:21.766242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" event={"ID":"f3d38f2b-c3d3-41d8-9129-7e6b6208655a","Type":"ContainerStarted","Data":"730af99ace2652021d4d7abedb363872bd1dddf6d8555299173135e8628effa5"} Jul 31 00:10:22.414019 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:22.413990 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ck7r_232e2b98-3c4e-415e-b538-6e350a6df518/node-ca/0.log" Jul 31 00:10:22.425887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:22.425857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:22.426072 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:22.425911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:22.426072 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:22.426034 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 00:10:22.426209 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:22.426106 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls podName:31f3ae79-4f9e-46cd-bd66-25e2179d6915 nodeName:}" failed. No retries permitted until 2026-07-31 00:10:26.426084542 +0000 UTC m=+235.949052278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r74xz" (UID: "31f3ae79-4f9e-46cd-bd66-25e2179d6915") : secret "insights-runtime-extractor-tls" not found Jul 31 00:10:22.426397 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:22.426377 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31f3ae79-4f9e-46cd-bd66-25e2179d6915-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:23.214231 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:23.214194 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87299_63977669-cfbc-4363-9091-f32b6585ae23/serve-healthcheck-canary/0.log" Jul 31 00:10:23.774640 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:23.774601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" event={"ID":"f3d38f2b-c3d3-41d8-9129-7e6b6208655a","Type":"ContainerStarted","Data":"da2474f52a89e53d2cbccfb6d7d8506a3715ffa44ce7449395c1c9d2b724e1da"} Jul 31 00:10:23.788824 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:23.788779 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-dt9fm" podStartSLOduration=1.910223432 podStartE2EDuration="3.788762792s" totalImagesPullingTime="1.87853936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:10:23.788642395 +0000 UTC m=+233.311610140" watchObservedRunningTime="2026-07-31 00:10:23.788762792 +0000 UTC m=+233.311730537" Jul 31 00:10:23.790617 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:23.790588 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:24.768364 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:24.768317 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:24.909850 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:24.909815 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:25.011311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:25.011281 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:25.111978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:25.111888 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:25.212393 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:25.212359 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:25.434223 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:25.434185 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:26.031488 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:26.031452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:26.457136 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:26.457072 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:26.457303 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:26.457243 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 00:10:26.457393 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:10:26.457317 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls podName:31f3ae79-4f9e-46cd-bd66-25e2179d6915 nodeName:}" failed. No retries permitted until 2026-07-31 00:10:34.457298779 +0000 UTC m=+243.980266507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r74xz" (UID: "31f3ae79-4f9e-46cd-bd66-25e2179d6915") : secret "insights-runtime-extractor-tls" not found Jul 31 00:10:26.652484 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:26.652454 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:27.038308 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:27.038269 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:27.231050 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:27.231015 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:27.835464 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:27.835425 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:28.260044 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:28.260010 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:28.439962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:28.439920 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:28.491766 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:28.491738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:28.635419 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:28.635334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:28.710674 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:28.710646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:28.832784 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:28.832753 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:29.037302 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:29.037268 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:29.139894 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:29.139864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:29.189887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:29.189857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:30.232855 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:30.232819 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:30.432956 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:30.432923 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:30.633246 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:30.633160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:30.830850 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:30.830819 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:31.033010 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:31.032977 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:31.236222 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:31.236182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:32.029281 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:32.029246 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:32.438618 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:32.438578 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:32.704848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:32.704765 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:32.834938 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:32.834897 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:33.031456 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:33.031375 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:34.237779 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.237738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:34.432970 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.432936 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:34.516766 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.516686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:34.519093 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.519068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31f3ae79-4f9e-46cd-bd66-25e2179d6915-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r74xz\" (UID: \"31f3ae79-4f9e-46cd-bd66-25e2179d6915\") " pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:34.560739 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.560709 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f672z\"" Jul 31 00:10:34.568732 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.568704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r74xz" Jul 31 00:10:34.636282 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.636229 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:34.693683 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.693645 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r74xz"] Jul 31 00:10:34.696594 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:10:34.696568 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f3ae79_4f9e_46cd_bd66_25e2179d6915.slice/crio-276c45f084488ee6d73a0f352b3962630482c1c4e3679a62d8feb6036b152589 WatchSource:0}: Error finding container 276c45f084488ee6d73a0f352b3962630482c1c4e3679a62d8feb6036b152589: Status 404 returned error can't find the container with id 276c45f084488ee6d73a0f352b3962630482c1c4e3679a62d8feb6036b152589 Jul 31 00:10:34.808938 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.808851 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r74xz" event={"ID":"31f3ae79-4f9e-46cd-bd66-25e2179d6915","Type":"ContainerStarted","Data":"96e7c5d9dc6e04ec2bffc991ab52ab7ba7b413eabbc1658edd1f0d8ba58142d8"} Jul 31 00:10:34.808938 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.808909 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r74xz" event={"ID":"31f3ae79-4f9e-46cd-bd66-25e2179d6915","Type":"ContainerStarted","Data":"276c45f084488ee6d73a0f352b3962630482c1c4e3679a62d8feb6036b152589"} Jul 31 00:10:34.856703 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:34.856672 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:35.034439 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:35.034408 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:35.432026 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:35.431996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:35.813773 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:35.813687 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r74xz" event={"ID":"31f3ae79-4f9e-46cd-bd66-25e2179d6915","Type":"ContainerStarted","Data":"44345d85e85217ab3b3e9a028ea12e943244abe1213ed206ab1fca3dfaed8483"} Jul 31 00:10:37.433998 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:37.433968 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:37.535502 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:37.535474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:37.725202 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:37.725082 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:37.822333 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:37.822296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r74xz" event={"ID":"31f3ae79-4f9e-46cd-bd66-25e2179d6915","Type":"ContainerStarted","Data":"7c3b5ba7e8149296d0a79cec9793dc0c39294966737d2ee9f28c101e19cab49d"} Jul 31 00:10:37.839258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:37.839206 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r74xz" podStartSLOduration=17.546535988 podStartE2EDuration="19.839190737s" totalImagesPullingTime="2.292654749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 00:10:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:10:37.839150679 +0000 UTC m=+247.362118433" watchObservedRunningTime="2026-07-31 00:10:37.839190737 +0000 UTC m=+247.362158475" Jul 31 00:10:38.124645 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:38.124522 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:38.171022 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:38.170994 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:38.304592 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:38.304558 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:38.355668 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:38.355636 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:38.706268 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:38.706235 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:38.764621 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:38.764590 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:38.917960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:38.917925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:39.318964 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:39.318926 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:39.718076 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:39.718025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:40.135354 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:40.135272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:40.233775 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:40.233738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:40.334043 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:40.334014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:41.764494 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:41.764457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:10:41.766828 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:41.766805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d-metrics-certs\") pod \"network-metrics-daemon-6xpcj\" (UID: \"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d\") " pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:10:41.945439 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:41.945402 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tbfb2\"" Jul 31 00:10:41.953533 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:41.953509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xpcj" Jul 31 00:10:42.077135 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:42.077093 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xpcj"] Jul 31 00:10:42.080403 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:10:42.080373 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7156ea2e_a762_4d4b_a0ec_1d70e1d47e0d.slice/crio-2991e7b5bbe7232f16a521b55ebb3658fcd83512ce760b0bed4188d000ac6199 WatchSource:0}: Error finding container 2991e7b5bbe7232f16a521b55ebb3658fcd83512ce760b0bed4188d000ac6199: Status 404 returned error can't find the container with id 2991e7b5bbe7232f16a521b55ebb3658fcd83512ce760b0bed4188d000ac6199 Jul 31 00:10:42.839663 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:42.839621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xpcj" event={"ID":"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d","Type":"ContainerStarted","Data":"2991e7b5bbe7232f16a521b55ebb3658fcd83512ce760b0bed4188d000ac6199"} Jul 31 00:10:43.844614 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:43.844577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xpcj" event={"ID":"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d","Type":"ContainerStarted","Data":"555ab8f1a30ae37a0ac903de883fb601edb081c74a2b01c50c0e840bee565df1"} Jul 31 00:10:43.844992 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:43.844631 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xpcj" event={"ID":"7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d","Type":"ContainerStarted","Data":"0a3eb23e47b342d6415d2094add6e4adc1e36572c58424d26c6dd8657fc8337b"} Jul 31 00:10:43.859209 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:43.859155 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6xpcj" podStartSLOduration=251.77082054 podStartE2EDuration="4m12.859136896s" totalImagesPullingTime="1.088316356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:10:43.858310208 +0000 UTC m=+253.381277952" watchObservedRunningTime="2026-07-31 00:10:43.859136896 +0000 UTC m=+253.382104632" Jul 31 00:10:43.935827 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:43.935797 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:45.074575 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:45.074513 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:45.931588 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:45.931555 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:47.935401 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:47.935360 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:47.979820 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:47.979787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:48.238947 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:48.238857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:48.339403 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:48.339367 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:49.637453 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:49.637422 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:49.736213 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:49.736174 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:49.808302 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:49.808270 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:49.860362 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:49.860316 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:50.057484 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:50.057436 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:50.331318 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:50.331231 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:50.387154 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:50.387095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:59.133784 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:59.133676 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:59.819349 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:59.819311 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:59.886723 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:59.886686 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:10:59.974012 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:10:59.973983 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:01.719652 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:01.719620 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:01.766170 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:01.766109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:01.865054 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:01.865025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:01.966429 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:01.966398 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:02.542586 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:02.542548 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:02.643168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:02.643132 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:02.792054 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:02.792023 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:03.735263 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.735217 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8f479cb88-jvr28"] Jul 31 00:11:03.738073 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.738048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.740952 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.740929 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 00:11:03.741113 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.740994 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 00:11:03.741113 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.741018 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 31 00:11:03.741113 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.741001 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 31 00:11:03.742249 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.742201 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 31 00:11:03.742249 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.742232 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 31 00:11:03.742428 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.742255 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 31 00:11:03.742428 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.742261 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c2ls4\"" Jul 31 00:11:03.747341 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.747320 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 31 00:11:03.747782 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.747761 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f479cb88-jvr28"] Jul 31 00:11:03.804174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.804114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-serving-cert\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.804174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.804177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-service-ca\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.804626 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.804199 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-config\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.804626 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.804233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-oauth-config\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.804626 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.804250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-oauth-serving-cert\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.804626 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.804295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wc6\" (UniqueName: \"kubernetes.io/projected/3beabded-ebaa-46f0-8d3a-d0125548ce3a-kube-api-access-97wc6\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.804626 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.804347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-trusted-ca-bundle\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.905598 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.905565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-serving-cert\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.905598 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.905596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-service-ca\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.905918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.905616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-config\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.905918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.905640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-oauth-config\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.905918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.905662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-oauth-serving-cert\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.905918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.905696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wc6\" (UniqueName: \"kubernetes.io/projected/3beabded-ebaa-46f0-8d3a-d0125548ce3a-kube-api-access-97wc6\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.905918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.905737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-trusted-ca-bundle\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.906463 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.906438 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-oauth-serving-cert\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.906568 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.906446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-config\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.906568 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.906496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-service-ca\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.906859 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.906839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-trusted-ca-bundle\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.908049 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.908020 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-oauth-config\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.908300 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.908282 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-serving-cert\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:03.914156 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:03.914132 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wc6\" (UniqueName: \"kubernetes.io/projected/3beabded-ebaa-46f0-8d3a-d0125548ce3a-kube-api-access-97wc6\") pod \"console-8f479cb88-jvr28\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:04.047491 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:04.047391 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:04.174346 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:04.174218 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f479cb88-jvr28"] Jul 31 00:11:04.176964 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:11:04.176931 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beabded_ebaa_46f0_8d3a_d0125548ce3a.slice/crio-e3d2cfa73ebf818009f25e8b6f5312edeecefcdcf49c99c2d3d415ad3e619568 WatchSource:0}: Error finding container e3d2cfa73ebf818009f25e8b6f5312edeecefcdcf49c99c2d3d415ad3e619568: Status 404 returned error can't find the container with id e3d2cfa73ebf818009f25e8b6f5312edeecefcdcf49c99c2d3d415ad3e619568 Jul 31 00:11:04.909937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:04.909869 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f479cb88-jvr28" event={"ID":"3beabded-ebaa-46f0-8d3a-d0125548ce3a","Type":"ContainerStarted","Data":"e3d2cfa73ebf818009f25e8b6f5312edeecefcdcf49c99c2d3d415ad3e619568"} Jul 31 00:11:04.964479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:04.964448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.170178 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.170076 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.190586 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.190550 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8"] Jul 31 00:11:05.193382 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.193155 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.196014 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.195987 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 31 00:11:05.196320 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.196085 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 31 00:11:05.197637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.197073 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 00:11:05.197637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.197316 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 00:11:05.197637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.197382 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 00:11:05.197637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.197328 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2cjh4\"" Jul 31 00:11:05.209364 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.209332 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n9qjm"] Jul 31 00:11:05.211779 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.211756 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8"] Jul 31 00:11:05.212052 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.212037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.215098 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.214999 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 00:11:05.215228 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.215220 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 00:11:05.215393 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.215366 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r9g7t\"" Jul 31 00:11:05.215646 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.215632 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 00:11:05.216684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.216630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzrj\" (UniqueName: \"kubernetes.io/projected/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-kube-api-access-pdzrj\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.216825 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.216808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.216967 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.216951 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.217102 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.217088 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.269437 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.269395 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.318068 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.317958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-root\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318068 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318068 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-textfile\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzrj\" (UniqueName: \"kubernetes.io/projected/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-kube-api-access-pdzrj\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-sys\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-metrics-client-ca\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-tls\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-wtmp\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318305 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92fmq\" (UniqueName: \"kubernetes.io/projected/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-kube-api-access-92fmq\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.318803 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.318803 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.318757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.318899 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:11:05.318869 2575 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 31 00:11:05.318948 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:11:05.318931 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-tls podName:4f6e14a0-b26d-4d2a-904c-f78cd4179cae nodeName:}" failed. No retries permitted until 2026-07-31 00:11:05.81891191 +0000 UTC m=+275.341879636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-f9mc8" (UID: "4f6e14a0-b26d-4d2a-904c-f78cd4179cae") : secret "openshift-state-metrics-tls" not found Jul 31 00:11:05.321676 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.321643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.327408 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.327381 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzrj\" (UniqueName: \"kubernetes.io/projected/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-kube-api-access-pdzrj\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.330648 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.330618 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.419071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92fmq\" (UniqueName: \"kubernetes.io/projected/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-kube-api-access-92fmq\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419295 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419295 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-root\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419295 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419295 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-root\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419508 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-textfile\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419508 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-sys\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419508 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-metrics-client-ca\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419657 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419528 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-sys\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419657 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-tls\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419657 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-wtmp\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419657 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-textfile\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419852 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:11:05.419743 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 31 00:11:05.419852 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-wtmp\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.419852 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:11:05.419800 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-tls podName:90c20e03-2d27-4fd7-b91a-0ab366cfcdcc nodeName:}" failed. No retries permitted until 2026-07-31 00:11:05.91978078 +0000 UTC m=+275.442748521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-tls") pod "node-exporter-n9qjm" (UID: "90c20e03-2d27-4fd7-b91a-0ab366cfcdcc") : secret "node-exporter-tls" not found Jul 31 00:11:05.419852 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.420055 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.419959 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-metrics-client-ca\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.421461 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.421413 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.427776 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.427751 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92fmq\" (UniqueName: \"kubernetes.io/projected/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-kube-api-access-92fmq\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.449929 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.449889 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.550852 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.550815 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.599998 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.599952 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.745869 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.745786 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.822876 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.822838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.826060 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.826028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f6e14a0-b26d-4d2a-904c-f78cd4179cae-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-f9mc8\" (UID: \"4f6e14a0-b26d-4d2a-904c-f78cd4179cae\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:05.872030 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.871992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:05.923640 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.923604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-tls\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:05.926298 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:05.926266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90c20e03-2d27-4fd7-b91a-0ab366cfcdcc-node-exporter-tls\") pod \"node-exporter-n9qjm\" (UID: \"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc\") " pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:06.112391 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:06.112310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" Jul 31 00:11:06.126590 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:06.126553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9qjm" Jul 31 00:11:06.186280 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:06.186233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:06.263425 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:06.263393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:06.389469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:06.389388 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:06.487553 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:06.487523 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:07.141963 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:11:07.141918 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c20e03_2d27_4fd7_b91a_0ab366cfcdcc.slice/crio-2f2fdcb24e0bef041e5c8adafe994c33f24eb66e7d85ecd22d09a55bacee41ca WatchSource:0}: Error finding container 2f2fdcb24e0bef041e5c8adafe994c33f24eb66e7d85ecd22d09a55bacee41ca: Status 404 returned error can't find the container with id 2f2fdcb24e0bef041e5c8adafe994c33f24eb66e7d85ecd22d09a55bacee41ca Jul 31 00:11:07.264718 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:07.264678 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8"] Jul 31 00:11:07.268080 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:11:07.268052 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6e14a0_b26d_4d2a_904c_f78cd4179cae.slice/crio-2c1619a14657ce9b4cc4df76574188d56605121b40dd0114fc9485a1e4b89028 WatchSource:0}: Error finding container 2c1619a14657ce9b4cc4df76574188d56605121b40dd0114fc9485a1e4b89028: Status 404 returned error can't find the container with id 2c1619a14657ce9b4cc4df76574188d56605121b40dd0114fc9485a1e4b89028 Jul 31 00:11:07.922805 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:07.922763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f479cb88-jvr28" event={"ID":"3beabded-ebaa-46f0-8d3a-d0125548ce3a","Type":"ContainerStarted","Data":"0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40"} Jul 31 00:11:07.924211 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:07.924184 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9qjm" event={"ID":"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc","Type":"ContainerStarted","Data":"2f2fdcb24e0bef041e5c8adafe994c33f24eb66e7d85ecd22d09a55bacee41ca"} Jul 31 00:11:07.926331 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:07.926294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" event={"ID":"4f6e14a0-b26d-4d2a-904c-f78cd4179cae","Type":"ContainerStarted","Data":"282b4d0d60dda3c200f6e75609266fbd1881d603d1ef197cb288a6015d5b583b"} Jul 31 00:11:07.926454 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:07.926357 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" event={"ID":"4f6e14a0-b26d-4d2a-904c-f78cd4179cae","Type":"ContainerStarted","Data":"688d6ffa28121c35fc65efee8d23a22e9cf85b6ff1198ab241e5189b521c1ba2"} Jul 31 00:11:07.926454 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:07.926383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" event={"ID":"4f6e14a0-b26d-4d2a-904c-f78cd4179cae","Type":"ContainerStarted","Data":"2c1619a14657ce9b4cc4df76574188d56605121b40dd0114fc9485a1e4b89028"} Jul 31 00:11:07.943240 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:07.943184 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8f479cb88-jvr28" podStartSLOduration=1.954000158 podStartE2EDuration="4.943163436s" totalImagesPullingTime="2.989163278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:11:07.941321242 +0000 UTC m=+277.464288987" watchObservedRunningTime="2026-07-31 00:11:07.943163436 +0000 UTC m=+277.466131180" Jul 31 00:11:08.124839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.124804 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:08.228464 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.228333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:08.325807 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.325769 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:08.376862 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.376821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:08.609609 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.609551 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:08.931212 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.931171 2575 generic.go:356] "Generic (PLEG): container finished" podID="90c20e03-2d27-4fd7-b91a-0ab366cfcdcc" containerID="16acbf53824330aca5e884c2f410de07c9c7b273a12d90b1eb269b714df8fb2c" exitCode=0 Jul 31 00:11:08.931404 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.931263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9qjm" event={"ID":"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc","Type":"ContainerDied","Data":"16acbf53824330aca5e884c2f410de07c9c7b273a12d90b1eb269b714df8fb2c"} Jul 31 00:11:08.933400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.933361 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" event={"ID":"4f6e14a0-b26d-4d2a-904c-f78cd4179cae","Type":"ContainerStarted","Data":"af4783bae2c886f44bd49caecf596c261c9202d4f00d32ac31413941a0338a7b"} Jul 31 00:11:08.966236 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:08.966173 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-f9mc8" podStartSLOduration=2.781214084 podStartE2EDuration="3.966149864s" totalImagesPullingTime="1.18493578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:11:08.964819174 +0000 UTC m=+278.487786968" watchObservedRunningTime="2026-07-31 00:11:08.966149864 +0000 UTC m=+278.489117610" Jul 31 00:11:09.366311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.366229 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:09.414834 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.414790 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:09.513895 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.513861 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f65b5c676-46bnn"] Jul 31 00:11:09.516846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.516830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.520455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.520429 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zbswh\"" Jul 31 00:11:09.520588 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.520477 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 31 00:11:09.520588 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.520424 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 31 00:11:09.520588 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.520516 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-94di123b452gp\"" Jul 31 00:11:09.520588 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.520429 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 31 00:11:09.520588 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.520479 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 31 00:11:09.525021 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.524997 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f65b5c676-46bnn"] Jul 31 00:11:09.553543 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.553503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/11d84d86-70c6-45c9-a3c4-aabaed72be9d-audit-log\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.553543 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.553549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-client-ca-bundle\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.553764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.553579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-secret-metrics-server-client-certs\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.553764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.553639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/11d84d86-70c6-45c9-a3c4-aabaed72be9d-metrics-server-audit-profiles\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.553764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.553670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-secret-metrics-server-tls\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.553764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.553695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncn9b\" (UniqueName: \"kubernetes.io/projected/11d84d86-70c6-45c9-a3c4-aabaed72be9d-kube-api-access-ncn9b\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.553764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.553725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d84d86-70c6-45c9-a3c4-aabaed72be9d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.654934 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.654905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/11d84d86-70c6-45c9-a3c4-aabaed72be9d-metrics-server-audit-profiles\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.654943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-secret-metrics-server-tls\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.654968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncn9b\" (UniqueName: \"kubernetes.io/projected/11d84d86-70c6-45c9-a3c4-aabaed72be9d-kube-api-access-ncn9b\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.654998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d84d86-70c6-45c9-a3c4-aabaed72be9d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655257 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.655083 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/11d84d86-70c6-45c9-a3c4-aabaed72be9d-audit-log\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655257 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.655144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-client-ca-bundle\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655257 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.655186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-secret-metrics-server-client-certs\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655527 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.655492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/11d84d86-70c6-45c9-a3c4-aabaed72be9d-audit-log\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655790 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.655763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d84d86-70c6-45c9-a3c4-aabaed72be9d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.655932 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.655913 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/11d84d86-70c6-45c9-a3c4-aabaed72be9d-metrics-server-audit-profiles\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.657690 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.657667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-secret-metrics-server-client-certs\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.657855 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.657825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-secret-metrics-server-tls\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.657923 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.657878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/11d84d86-70c6-45c9-a3c4-aabaed72be9d-client-ca-bundle\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.663456 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.663428 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncn9b\" (UniqueName: \"kubernetes.io/projected/11d84d86-70c6-45c9-a3c4-aabaed72be9d-kube-api-access-ncn9b\") pod \"metrics-server-f65b5c676-46bnn\" (UID: \"11d84d86-70c6-45c9-a3c4-aabaed72be9d\") " pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.811764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.811711 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:09.826708 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.826669 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:09.912580 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.912545 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:09.943132 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.943080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9qjm" event={"ID":"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc","Type":"ContainerStarted","Data":"893438b918491071daa1a6f390444ad56d26a08ab1caf1c7ca7de0134cca824a"} Jul 31 00:11:09.943294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.943182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9qjm" event={"ID":"90c20e03-2d27-4fd7-b91a-0ab366cfcdcc","Type":"ContainerStarted","Data":"66b9e2245920474f3ec96635587a06ad1ae82029e958f5505fd0aa4996b64782"} Jul 31 00:11:09.962709 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.962688 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f65b5c676-46bnn"] Jul 31 00:11:09.964242 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.964193 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n9qjm" podStartSLOduration=4.065605817 podStartE2EDuration="4.964174621s" totalImagesPullingTime="825.912893ms" totalInitContainerRuntime="72.655911ms" isStatefulPod=true podCreationTimestamp="2026-07-31 00:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:11:09.963748786 +0000 UTC m=+279.486716556" watchObservedRunningTime="2026-07-31 00:11:09.964174621 +0000 UTC m=+279.487142368" Jul 31 00:11:09.965566 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:11:09.965541 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d84d86_70c6_45c9_a3c4_aabaed72be9d.slice/crio-3b014ae9c6650ffae0df37c75ba8e19619f938bc33198f34a83c05e16c02ea78 WatchSource:0}: Error finding container 3b014ae9c6650ffae0df37c75ba8e19619f938bc33198f34a83c05e16c02ea78: Status 404 returned error can't find the container with id 3b014ae9c6650ffae0df37c75ba8e19619f938bc33198f34a83c05e16c02ea78 Jul 31 00:11:09.997643 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.997605 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p"] Jul 31 00:11:09.998338 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:09.998301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:10.002263 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.002244 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.004651 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.004628 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 31 00:11:10.004791 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.004719 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7qr8v\"" Jul 31 00:11:10.008223 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.008196 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p"] Jul 31 00:11:10.058110 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.058072 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71ba5396-440c-4776-9dc6-c22345849b06-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-xsv8p\" (UID: \"71ba5396-440c-4776-9dc6-c22345849b06\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.058322 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.058113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4p9q\" (UniqueName: \"kubernetes.io/projected/71ba5396-440c-4776-9dc6-c22345849b06-kube-api-access-j4p9q\") pod \"monitoring-plugin-6dffbbd67-xsv8p\" (UID: \"71ba5396-440c-4776-9dc6-c22345849b06\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.115818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.115788 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:10.158709 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.158634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71ba5396-440c-4776-9dc6-c22345849b06-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-xsv8p\" (UID: \"71ba5396-440c-4776-9dc6-c22345849b06\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.158709 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.158676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4p9q\" (UniqueName: \"kubernetes.io/projected/71ba5396-440c-4776-9dc6-c22345849b06-kube-api-access-j4p9q\") pod \"monitoring-plugin-6dffbbd67-xsv8p\" (UID: \"71ba5396-440c-4776-9dc6-c22345849b06\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.158864 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:11:10.158778 2575 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 31 00:11:10.158864 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:11:10.158841 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71ba5396-440c-4776-9dc6-c22345849b06-monitoring-plugin-cert podName:71ba5396-440c-4776-9dc6-c22345849b06 nodeName:}" failed. No retries permitted until 2026-07-31 00:11:10.658825467 +0000 UTC m=+280.181793189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/71ba5396-440c-4776-9dc6-c22345849b06-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-xsv8p" (UID: "71ba5396-440c-4776-9dc6-c22345849b06") : secret "monitoring-plugin-cert" not found Jul 31 00:11:10.167335 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.167311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4p9q\" (UniqueName: \"kubernetes.io/projected/71ba5396-440c-4776-9dc6-c22345849b06-kube-api-access-j4p9q\") pod \"monitoring-plugin-6dffbbd67-xsv8p\" (UID: \"71ba5396-440c-4776-9dc6-c22345849b06\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.487692 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.487662 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:10.537638 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.537600 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:10.587941 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.587906 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:10.663267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.663187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71ba5396-440c-4776-9dc6-c22345849b06-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-xsv8p\" (UID: \"71ba5396-440c-4776-9dc6-c22345849b06\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.666660 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.666627 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/71ba5396-440c-4776-9dc6-c22345849b06-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-xsv8p\" (UID: \"71ba5396-440c-4776-9dc6-c22345849b06\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.913666 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.913621 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:10.948565 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:10.948530 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" event={"ID":"11d84d86-70c6-45c9-a3c4-aabaed72be9d","Type":"ContainerStarted","Data":"3b014ae9c6650ffae0df37c75ba8e19619f938bc33198f34a83c05e16c02ea78"} Jul 31 00:11:11.340879 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:11.340844 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p"] Jul 31 00:11:11.344415 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:11:11.344386 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ba5396_440c_4776_9dc6_c22345849b06.slice/crio-effc0137859ead869c2d99eb7259569605e435349324c291ed6da19531dc43df WatchSource:0}: Error finding container effc0137859ead869c2d99eb7259569605e435349324c291ed6da19531dc43df: Status 404 returned error can't find the container with id effc0137859ead869c2d99eb7259569605e435349324c291ed6da19531dc43df Jul 31 00:11:11.396079 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:11.396041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:11.748806 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:11.748769 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f479cb88-jvr28"] Jul 31 00:11:11.954255 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:11.954214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" event={"ID":"11d84d86-70c6-45c9-a3c4-aabaed72be9d","Type":"ContainerStarted","Data":"2daee97a736bf150feb2aed3e02027578a4a4808a79c16c188ad232fcb25e0c8"} Jul 31 00:11:11.955382 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:11.955349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" event={"ID":"71ba5396-440c-4776-9dc6-c22345849b06","Type":"ContainerStarted","Data":"effc0137859ead869c2d99eb7259569605e435349324c291ed6da19531dc43df"} Jul 31 00:11:12.558799 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:11:12.558747 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gtc62" podUID="7aa00574-87fc-449d-a891-4b43e2f0d014" Jul 31 00:11:12.959231 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:12.959196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtc62" Jul 31 00:11:13.963956 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:13.963919 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" event={"ID":"71ba5396-440c-4776-9dc6-c22345849b06","Type":"ContainerStarted","Data":"f8230ce2b4f2d560730ae0882a25709f1d5e053ca9b0cefa779376ae5357b24d"} Jul 31 00:11:13.964375 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:13.964154 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:13.969017 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:13.968993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" Jul 31 00:11:13.978586 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:13.978531 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-xsv8p" podStartSLOduration=2.921641018 podStartE2EDuration="4.978518927s" totalImagesPullingTime="2.056877909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:11:13.978077904 +0000 UTC m=+283.501045672" watchObservedRunningTime="2026-07-31 00:11:13.978518927 +0000 UTC m=+283.501486672" Jul 31 00:11:13.979234 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:13.979208 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" podStartSLOduration=3.699046064 podStartE2EDuration="4.979198397s" totalImagesPullingTime="1.280152333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:11:11.969292843 +0000 UTC m=+281.492260589" watchObservedRunningTime="2026-07-31 00:11:13.979198397 +0000 UTC m=+283.502166141" Jul 31 00:11:14.048157 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:14.048100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:14.104764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:14.104737 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:14.150230 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:14.150204 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:15.798381 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:15.798354 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:15.810461 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:15.810425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:11:15.812673 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:15.812646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7aa00574-87fc-449d-a891-4b43e2f0d014-metrics-tls\") pod \"dns-default-gtc62\" (UID: \"7aa00574-87fc-449d-a891-4b43e2f0d014\") " pod="openshift-dns/dns-default-gtc62" Jul 31 00:11:15.850642 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:15.850609 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:15.962204 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:15.962179 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nnjn4\"" Jul 31 00:11:15.969806 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:15.969780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtc62" Jul 31 00:11:16.096184 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:16.096161 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gtc62"] Jul 31 00:11:16.098567 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:11:16.098533 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa00574_87fc_449d_a891_4b43e2f0d014.slice/crio-3c137b756ac73198a1e5aefa86fa682c96688a53b678fd7ae38e3191f1c7b8bf WatchSource:0}: Error finding container 3c137b756ac73198a1e5aefa86fa682c96688a53b678fd7ae38e3191f1c7b8bf: Status 404 returned error can't find the container with id 3c137b756ac73198a1e5aefa86fa682c96688a53b678fd7ae38e3191f1c7b8bf Jul 31 00:11:16.975756 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:16.975715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtc62" event={"ID":"7aa00574-87fc-449d-a891-4b43e2f0d014","Type":"ContainerStarted","Data":"3c137b756ac73198a1e5aefa86fa682c96688a53b678fd7ae38e3191f1c7b8bf"} Jul 31 00:11:17.980957 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:17.980915 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtc62" event={"ID":"7aa00574-87fc-449d-a891-4b43e2f0d014","Type":"ContainerStarted","Data":"f657d55bb635303134451879a7f056908b85602310c93387030a33bef908cc27"} Jul 31 00:11:17.981373 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:17.980975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtc62" event={"ID":"7aa00574-87fc-449d-a891-4b43e2f0d014","Type":"ContainerStarted","Data":"759175e64fae98ffb8dd2d0cbe17ccfe1e772172ad093eb7176d49d8624eae9c"} Jul 31 00:11:17.981373 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:17.981013 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gtc62" Jul 31 00:11:17.996952 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:17.996856 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gtc62" podStartSLOduration=251.349481629 podStartE2EDuration="4m12.996837645s" totalImagesPullingTime="1.647356016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:11:17.99571078 +0000 UTC m=+287.518678544" watchObservedRunningTime="2026-07-31 00:11:17.996837645 +0000 UTC m=+287.519805390" Jul 31 00:11:19.165551 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:19.165519 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:19.269509 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:19.269475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:22.958998 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:22.958962 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:23.060481 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:23.060447 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:23.159794 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:23.159763 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:24.867598 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:24.867570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:24.925550 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:24.925521 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:27.987711 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:27.987677 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gtc62" Jul 31 00:11:29.106833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:29.106801 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:29.156463 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:29.156432 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:29.827769 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:29.827732 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:29.827950 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:29.827809 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:30.880644 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:30.880612 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 00:11:32.930072 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:32.930040 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:36.768094 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:36.768035 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8f479cb88-jvr28" podUID="3beabded-ebaa-46f0-8d3a-d0125548ce3a" containerName="console" containerID="cri-o://0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40" gracePeriod=15 Jul 31 00:11:37.012758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.012730 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:37.049966 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.049880 2575 generic.go:356] "Generic (PLEG): container finished" podID="3beabded-ebaa-46f0-8d3a-d0125548ce3a" containerID="0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40" exitCode=0 Jul 31 00:11:37.049966 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.049921 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f479cb88-jvr28" Jul 31 00:11:37.050185 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.049961 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f479cb88-jvr28" event={"ID":"3beabded-ebaa-46f0-8d3a-d0125548ce3a","Type":"ContainerDied","Data":"0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40"} Jul 31 00:11:37.050185 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.050007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f479cb88-jvr28" event={"ID":"3beabded-ebaa-46f0-8d3a-d0125548ce3a","Type":"ContainerDied","Data":"e3d2cfa73ebf818009f25e8b6f5312edeecefcdcf49c99c2d3d415ad3e619568"} Jul 31 00:11:37.050185 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.050028 2575 scope.go:117] "RemoveContainer" containerID="0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40" Jul 31 00:11:37.057938 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.057918 2575 scope.go:117] "RemoveContainer" containerID="0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40" Jul 31 00:11:37.058343 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:11:37.058322 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40\": container with ID starting with 0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40 not found: ID does not exist" containerID="0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40" Jul 31 00:11:37.058401 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.058357 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40"} err="failed to get container status \"0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40\": rpc error: code = NotFound desc = could not find container \"0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40\": container with ID starting with 0dc59da68db98d5c64cb851ea20007ec4aa4ac98a50b30221b4e3b2a5ddfaf40 not found: ID does not exist" Jul 31 00:11:37.083833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.083799 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-oauth-serving-cert\") pod \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " Jul 31 00:11:37.083978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.083854 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-oauth-config\") pod \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " Jul 31 00:11:37.083978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.083875 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-service-ca\") pod \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " Jul 31 00:11:37.083978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.083928 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-serving-cert\") pod \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " Jul 31 00:11:37.083978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.083946 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-config\") pod \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " Jul 31 00:11:37.083978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.083965 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97wc6\" (UniqueName: \"kubernetes.io/projected/3beabded-ebaa-46f0-8d3a-d0125548ce3a-kube-api-access-97wc6\") pod \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " Jul 31 00:11:37.083978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.083979 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-trusted-ca-bundle\") pod \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\" (UID: \"3beabded-ebaa-46f0-8d3a-d0125548ce3a\") " Jul 31 00:11:37.084394 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.084358 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-service-ca" (OuterVolumeSpecName: "service-ca") pod "3beabded-ebaa-46f0-8d3a-d0125548ce3a" (UID: "3beabded-ebaa-46f0-8d3a-d0125548ce3a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:11:37.084499 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.084478 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-config" (OuterVolumeSpecName: "console-config") pod "3beabded-ebaa-46f0-8d3a-d0125548ce3a" (UID: "3beabded-ebaa-46f0-8d3a-d0125548ce3a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:11:37.084616 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.084588 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3beabded-ebaa-46f0-8d3a-d0125548ce3a" (UID: "3beabded-ebaa-46f0-8d3a-d0125548ce3a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:11:37.084681 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.084609 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3beabded-ebaa-46f0-8d3a-d0125548ce3a" (UID: "3beabded-ebaa-46f0-8d3a-d0125548ce3a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:11:37.086352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.086325 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beabded-ebaa-46f0-8d3a-d0125548ce3a-kube-api-access-97wc6" (OuterVolumeSpecName: "kube-api-access-97wc6") pod "3beabded-ebaa-46f0-8d3a-d0125548ce3a" (UID: "3beabded-ebaa-46f0-8d3a-d0125548ce3a"). InnerVolumeSpecName "kube-api-access-97wc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:11:37.086440 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.086395 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3beabded-ebaa-46f0-8d3a-d0125548ce3a" (UID: "3beabded-ebaa-46f0-8d3a-d0125548ce3a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:11:37.086440 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.086423 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3beabded-ebaa-46f0-8d3a-d0125548ce3a" (UID: "3beabded-ebaa-46f0-8d3a-d0125548ce3a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:11:37.101653 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.101620 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:37.149072 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.149039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:37.184954 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.184925 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-serving-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:11:37.185051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.184957 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-config\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:11:37.185051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.184971 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97wc6\" (UniqueName: \"kubernetes.io/projected/3beabded-ebaa-46f0-8d3a-d0125548ce3a-kube-api-access-97wc6\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:11:37.185051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.184986 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-trusted-ca-bundle\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:11:37.185051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.184998 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-oauth-serving-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:11:37.185051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.185012 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beabded-ebaa-46f0-8d3a-d0125548ce3a-console-oauth-config\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:11:37.185051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.185025 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beabded-ebaa-46f0-8d3a-d0125548ce3a-service-ca\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:11:37.373124 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.373082 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f479cb88-jvr28"] Jul 31 00:11:37.374864 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:37.374840 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8f479cb88-jvr28"] Jul 31 00:11:39.047662 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:39.047632 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3beabded-ebaa-46f0-8d3a-d0125548ce3a" path="/var/lib/kubelet/pods/3beabded-ebaa-46f0-8d3a-d0125548ce3a/volumes" Jul 31 00:11:44.112246 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:44.112206 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:44.160674 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:44.160625 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:45.505945 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:45.505900 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" podUID="7a288f42-eede-4280-aa3d-c91b26aee6a9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 00:11:46.105513 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:46.105474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:11:49.832580 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:49.832544 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:49.836376 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:49.836350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f65b5c676-46bnn" Jul 31 00:11:55.506798 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:11:55.506747 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" podUID="7a288f42-eede-4280-aa3d-c91b26aee6a9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 00:12:05.505062 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:05.505022 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" podUID="7a288f42-eede-4280-aa3d-c91b26aee6a9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 31 00:12:05.505548 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:05.505095 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" Jul 31 00:12:05.505590 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:05.505572 2575 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"702414f6ba60d8e44733d2335c11764d444eaea72d3a68be5c2a634d68e4d8f5"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 31 00:12:05.505628 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:05.505610 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" podUID="7a288f42-eede-4280-aa3d-c91b26aee6a9" containerName="service-proxy" containerID="cri-o://702414f6ba60d8e44733d2335c11764d444eaea72d3a68be5c2a634d68e4d8f5" gracePeriod=30 Jul 31 00:12:06.155694 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:06.155661 2575 generic.go:356] "Generic (PLEG): container finished" podID="7a288f42-eede-4280-aa3d-c91b26aee6a9" containerID="702414f6ba60d8e44733d2335c11764d444eaea72d3a68be5c2a634d68e4d8f5" exitCode=2 Jul 31 00:12:06.155861 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:06.155723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" event={"ID":"7a288f42-eede-4280-aa3d-c91b26aee6a9","Type":"ContainerDied","Data":"702414f6ba60d8e44733d2335c11764d444eaea72d3a68be5c2a634d68e4d8f5"} Jul 31 00:12:06.155861 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:06.155756 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ccfb6ff45-ssvhl" event={"ID":"7a288f42-eede-4280-aa3d-c91b26aee6a9","Type":"ContainerStarted","Data":"dec9f18efaa89527ad88693d0313a8230a1b8723780442df437cd7e313cf3e8b"} Jul 31 00:12:15.599335 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:15.599295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:15.651847 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:15.651813 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:24.375421 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:24.375326 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:25.265756 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:25.265724 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:32.974841 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:32.974806 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:33.700208 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.700172 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79986c59b6-w2p4c"] Jul 31 00:12:33.700465 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.700453 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beabded-ebaa-46f0-8d3a-d0125548ce3a" containerName="console" Jul 31 00:12:33.700509 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.700467 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beabded-ebaa-46f0-8d3a-d0125548ce3a" containerName="console" Jul 31 00:12:33.700543 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.700516 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beabded-ebaa-46f0-8d3a-d0125548ce3a" containerName="console" Jul 31 00:12:33.703456 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.703436 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.705999 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.705974 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 00:12:33.707086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.707055 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 31 00:12:33.707192 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.707138 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 31 00:12:33.707277 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.707260 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 31 00:12:33.707406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.707380 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c2ls4\"" Jul 31 00:12:33.707525 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.707440 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 31 00:12:33.707525 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.707470 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 00:12:33.707525 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.707493 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 31 00:12:33.711693 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.711669 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 31 00:12:33.713958 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.713936 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79986c59b6-w2p4c"] Jul 31 00:12:33.751595 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.751561 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:33.795919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.795872 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:33.797982 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.797961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmn7t\" (UniqueName: \"kubernetes.io/projected/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-kube-api-access-gmn7t\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.798056 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.797996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-oauth-config\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.798056 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.798034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-config\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.798180 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.798147 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-oauth-serving-cert\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.798236 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.798190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-trusted-ca-bundle\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.798284 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.798236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-service-ca\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.798320 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.798301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-serving-cert\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.897948 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.897915 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:33.899025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-service-ca\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.899087 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-serving-cert\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.899087 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmn7t\" (UniqueName: \"kubernetes.io/projected/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-kube-api-access-gmn7t\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.899193 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-oauth-config\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.899258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-config\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.899317 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-oauth-serving-cert\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.899366 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-trusted-ca-bundle\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.899786 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899761 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-service-ca\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.899913 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.899877 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-config\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.900023 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.900003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-oauth-serving-cert\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.900463 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.900441 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-trusted-ca-bundle\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.901596 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.901575 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-oauth-config\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.901714 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.901698 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-serving-cert\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.907294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.907257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmn7t\" (UniqueName: \"kubernetes.io/projected/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-kube-api-access-gmn7t\") pod \"console-79986c59b6-w2p4c\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:33.999666 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:33.999531 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:34.013935 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:34.013901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:34.144271 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:34.144237 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79986c59b6-w2p4c"] Jul 31 00:12:34.147295 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:12:34.147262 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2102ea_0191_4f85_bdca_5f2d6cfc73bc.slice/crio-86f6285a7450bfae7e31d47b0a143691afd9b0d19c38d9160adc03e90f79397a WatchSource:0}: Error finding container 86f6285a7450bfae7e31d47b0a143691afd9b0d19c38d9160adc03e90f79397a: Status 404 returned error can't find the container with id 86f6285a7450bfae7e31d47b0a143691afd9b0d19c38d9160adc03e90f79397a Jul 31 00:12:34.253224 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:34.253113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79986c59b6-w2p4c" event={"ID":"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc","Type":"ContainerStarted","Data":"840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4"} Jul 31 00:12:34.253224 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:34.253189 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79986c59b6-w2p4c" event={"ID":"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc","Type":"ContainerStarted","Data":"86f6285a7450bfae7e31d47b0a143691afd9b0d19c38d9160adc03e90f79397a"} Jul 31 00:12:34.269684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:34.269634 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79986c59b6-w2p4c" podStartSLOduration=1.269615769 podStartE2EDuration="1.269615769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:12:34.268347807 +0000 UTC m=+363.791315572" watchObservedRunningTime="2026-07-31 00:12:34.269615769 +0000 UTC m=+363.792583514" Jul 31 00:12:34.496681 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:34.496646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:44.015036 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:44.014991 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:44.015526 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:44.015075 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:44.019885 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:44.019862 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:44.290252 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:44.290161 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:12:44.438880 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:44.438846 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:44.492459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:44.492419 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:12:44.545085 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:12:44.544984 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:14.382424 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:14.382381 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:14.430651 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:14.430613 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:39.603159 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:39.603106 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:39.655322 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:39.655291 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:40.176500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.176470 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-579f985bf4-jxjl4"] Jul 31 00:13:40.180709 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.180673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.192039 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.192014 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579f985bf4-jxjl4"] Jul 31 00:13:40.209965 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.209932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-serving-cert\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.210149 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.209978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-trusted-ca-bundle\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.210149 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.210034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-service-ca\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.210149 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.210094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-config\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.210149 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.210111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-oauth-serving-cert\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.210149 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.210149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrrbp\" (UniqueName: \"kubernetes.io/projected/1fd40b52-5468-4fc1-8c53-18a6ea26a963-kube-api-access-wrrbp\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.210340 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.210166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-oauth-config\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.226721 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.226692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:40.310758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.310729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-serving-cert\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.310937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.310777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-trusted-ca-bundle\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.310937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.310808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-service-ca\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.310937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.310865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-config\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.310937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.310887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-oauth-serving-cert\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.310937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.310916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrrbp\" (UniqueName: \"kubernetes.io/projected/1fd40b52-5468-4fc1-8c53-18a6ea26a963-kube-api-access-wrrbp\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.311202 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.310943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-oauth-config\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.311660 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.311621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-service-ca\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.311772 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.311717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-oauth-serving-cert\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.311841 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.311818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-trusted-ca-bundle\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.311900 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.311823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-config\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.313485 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.313461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-serving-cert\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.313485 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.313479 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-oauth-config\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.318916 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.318890 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrrbp\" (UniqueName: \"kubernetes.io/projected/1fd40b52-5468-4fc1-8c53-18a6ea26a963-kube-api-access-wrrbp\") pod \"console-579f985bf4-jxjl4\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.493353 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.493259 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:40.621030 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:40.620995 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579f985bf4-jxjl4"] Jul 31 00:13:40.624174 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:13:40.624144 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd40b52_5468_4fc1_8c53_18a6ea26a963.slice/crio-baf152bdfc297c914a27ad39dc223c14a9e29b642424100a722e4544892862a0 WatchSource:0}: Error finding container baf152bdfc297c914a27ad39dc223c14a9e29b642424100a722e4544892862a0: Status 404 returned error can't find the container with id baf152bdfc297c914a27ad39dc223c14a9e29b642424100a722e4544892862a0 Jul 31 00:13:41.488090 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:41.488057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579f985bf4-jxjl4" event={"ID":"1fd40b52-5468-4fc1-8c53-18a6ea26a963","Type":"ContainerStarted","Data":"90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05"} Jul 31 00:13:41.488292 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:41.488110 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579f985bf4-jxjl4" event={"ID":"1fd40b52-5468-4fc1-8c53-18a6ea26a963","Type":"ContainerStarted","Data":"baf152bdfc297c914a27ad39dc223c14a9e29b642424100a722e4544892862a0"} Jul 31 00:13:41.505277 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:41.505227 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-579f985bf4-jxjl4" podStartSLOduration=1.505210546 podStartE2EDuration="1.505210546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:13:41.504098104 +0000 UTC m=+431.027065902" watchObservedRunningTime="2026-07-31 00:13:41.505210546 +0000 UTC m=+431.028178290" Jul 31 00:13:50.493978 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:50.493938 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:50.494526 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:50.494032 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:50.498878 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:50.498853 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:50.525041 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:50.525013 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:13:50.567266 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:50.567236 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79986c59b6-w2p4c"] Jul 31 00:13:50.696361 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:50.696317 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:50.796145 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:50.796040 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:50.847903 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:50.847869 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:51.572363 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:51.572329 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:51.673024 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:51.672979 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:59.112328 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:59.112243 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:59.214534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:59.214491 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:13:59.842282 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:13:59.842246 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:14:15.586875 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.586829 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79986c59b6-w2p4c" podUID="6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" containerName="console" containerID="cri-o://840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4" gracePeriod=15 Jul 31 00:14:15.823555 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.823525 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:14:15.899535 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899440 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-service-ca\") pod \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " Jul 31 00:14:15.899535 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899480 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-trusted-ca-bundle\") pod \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " Jul 31 00:14:15.899535 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899513 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmn7t\" (UniqueName: \"kubernetes.io/projected/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-kube-api-access-gmn7t\") pod \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " Jul 31 00:14:15.899535 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899528 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-config\") pod \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " Jul 31 00:14:15.899866 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899575 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-serving-cert\") pod \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " Jul 31 00:14:15.899866 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899620 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-oauth-config\") pod \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " Jul 31 00:14:15.899866 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899682 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-oauth-serving-cert\") pod \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\" (UID: \"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc\") " Jul 31 00:14:15.900021 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899968 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-service-ca" (OuterVolumeSpecName: "service-ca") pod "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" (UID: "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:14:15.900021 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.899989 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-config" (OuterVolumeSpecName: "console-config") pod "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" (UID: "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:14:15.900166 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.900005 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" (UID: "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:14:15.900216 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.900204 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" (UID: "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:14:15.901815 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.901787 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" (UID: "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:14:15.902201 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.902180 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-kube-api-access-gmn7t" (OuterVolumeSpecName: "kube-api-access-gmn7t") pod "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" (UID: "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc"). InnerVolumeSpecName "kube-api-access-gmn7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:14:15.902283 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:15.902229 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" (UID: "6d2102ea-0191-4f85-bdca-5f2d6cfc73bc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:14:16.000534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.000495 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-service-ca\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:14:16.000534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.000527 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-trusted-ca-bundle\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:14:16.000534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.000538 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmn7t\" (UniqueName: \"kubernetes.io/projected/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-kube-api-access-gmn7t\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:14:16.000534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.000547 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-config\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:14:16.000802 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.000556 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-serving-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:14:16.000802 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.000565 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-console-oauth-config\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:14:16.000802 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.000575 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc-oauth-serving-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:14:16.611174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.611138 2575 generic.go:356] "Generic (PLEG): container finished" podID="6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" containerID="840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4" exitCode=0 Jul 31 00:14:16.611617 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.611182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79986c59b6-w2p4c" event={"ID":"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc","Type":"ContainerDied","Data":"840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4"} Jul 31 00:14:16.611617 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.611221 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79986c59b6-w2p4c" event={"ID":"6d2102ea-0191-4f85-bdca-5f2d6cfc73bc","Type":"ContainerDied","Data":"86f6285a7450bfae7e31d47b0a143691afd9b0d19c38d9160adc03e90f79397a"} Jul 31 00:14:16.611617 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.611240 2575 scope.go:117] "RemoveContainer" containerID="840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4" Jul 31 00:14:16.611617 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.611185 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79986c59b6-w2p4c" Jul 31 00:14:16.619625 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.619600 2575 scope.go:117] "RemoveContainer" containerID="840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4" Jul 31 00:14:16.619934 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:14:16.619912 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4\": container with ID starting with 840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4 not found: ID does not exist" containerID="840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4" Jul 31 00:14:16.620012 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.619942 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4"} err="failed to get container status \"840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4\": rpc error: code = NotFound desc = could not find container \"840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4\": container with ID starting with 840db5f40c55aaede84115893d0ae95615fb99bbdb3174fa04fee26717d36dd4 not found: ID does not exist" Jul 31 00:14:16.632431 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.632399 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79986c59b6-w2p4c"] Jul 31 00:14:16.635557 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:16.635474 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79986c59b6-w2p4c"] Jul 31 00:14:17.047970 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:17.047924 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" path="/var/lib/kubelet/pods/6d2102ea-0191-4f85-bdca-5f2d6cfc73bc/volumes" Jul 31 00:14:34.980322 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:34.980279 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:14:35.077510 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:35.077477 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:14:56.479536 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:56.479501 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:14:57.314833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:14:57.314792 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:00.420737 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:00.420707 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:00.523952 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:00.523914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:02.175237 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:02.175205 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:02.635641 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:02.635555 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:02.762751 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:02.762710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:04.113788 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.113749 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:04.477172 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.477104 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:04.543481 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.543447 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-c4h68"] Jul 31 00:15:04.543714 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.543702 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" containerName="console" Jul 31 00:15:04.543768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.543715 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" containerName="console" Jul 31 00:15:04.543768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.543760 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2102ea-0191-4f85-bdca-5f2d6cfc73bc" containerName="console" Jul 31 00:15:04.546568 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.546552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" Jul 31 00:15:04.548910 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.548889 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 31 00:15:04.549025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.548888 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2ct9w\"" Jul 31 00:15:04.550021 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.550004 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 31 00:15:04.554783 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.554762 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-c4h68"] Jul 31 00:15:04.578078 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.578045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f412121-5e04-421b-a310-86deac59adef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-c4h68\" (UID: \"8f412121-5e04-421b-a310-86deac59adef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" Jul 31 00:15:04.578251 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.578109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqp8p\" (UniqueName: \"kubernetes.io/projected/8f412121-5e04-421b-a310-86deac59adef-kube-api-access-kqp8p\") pod \"cert-manager-cainjector-559cdd8c87-c4h68\" (UID: \"8f412121-5e04-421b-a310-86deac59adef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" Jul 31 00:15:04.679411 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.679366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f412121-5e04-421b-a310-86deac59adef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-c4h68\" (UID: \"8f412121-5e04-421b-a310-86deac59adef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" Jul 31 00:15:04.679602 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.679445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqp8p\" (UniqueName: \"kubernetes.io/projected/8f412121-5e04-421b-a310-86deac59adef-kube-api-access-kqp8p\") pod \"cert-manager-cainjector-559cdd8c87-c4h68\" (UID: \"8f412121-5e04-421b-a310-86deac59adef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" Jul 31 00:15:04.688032 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.687996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqp8p\" (UniqueName: \"kubernetes.io/projected/8f412121-5e04-421b-a310-86deac59adef-kube-api-access-kqp8p\") pod \"cert-manager-cainjector-559cdd8c87-c4h68\" (UID: \"8f412121-5e04-421b-a310-86deac59adef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" Jul 31 00:15:04.688032 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.688030 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f412121-5e04-421b-a310-86deac59adef-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-c4h68\" (UID: \"8f412121-5e04-421b-a310-86deac59adef\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" Jul 31 00:15:04.856331 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.856242 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" Jul 31 00:15:04.984071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.984034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-c4h68"] Jul 31 00:15:04.986770 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:15:04.986738 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f412121_5e04_421b_a310_86deac59adef.slice/crio-5118e767acbdcb79a2aa96fa95195485eaf35bcf25c13c63c73516bd1fea5113 WatchSource:0}: Error finding container 5118e767acbdcb79a2aa96fa95195485eaf35bcf25c13c63c73516bd1fea5113: Status 404 returned error can't find the container with id 5118e767acbdcb79a2aa96fa95195485eaf35bcf25c13c63c73516bd1fea5113 Jul 31 00:15:04.988634 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:04.988614 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 00:15:05.786721 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:05.786685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" event={"ID":"8f412121-5e04-421b-a310-86deac59adef","Type":"ContainerStarted","Data":"5118e767acbdcb79a2aa96fa95195485eaf35bcf25c13c63c73516bd1fea5113"} Jul 31 00:15:07.083362 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:07.083301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:07.139855 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:07.139811 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:08.800168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:08.800134 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" event={"ID":"8f412121-5e04-421b-a310-86deac59adef","Type":"ContainerStarted","Data":"72f87eb809062a34c1ac25bb472b601bc49684485dd105290b8da2a78510e517"} Jul 31 00:15:08.818005 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:08.817947 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c4h68" podStartSLOduration=1.428063682 podStartE2EDuration="4.817928615s" totalImagesPullingTime="3.389864933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:15:08.817081617 +0000 UTC m=+518.340049368" watchObservedRunningTime="2026-07-31 00:15:08.817928615 +0000 UTC m=+518.340896359" Jul 31 00:15:09.782904 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:09.782867 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:09.931033 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:09.930996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:12.914527 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:12.914485 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:16.664722 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:16.664686 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:19.300221 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:19.300182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:19.398053 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:19.398022 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:21.114472 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:21.114426 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:21.162633 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:21.162603 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:25.095412 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:25.095334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:25.199498 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:25.199470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:27.295563 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:27.295531 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:42.512131 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:42.512083 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:42.622834 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:42.622804 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:43.116807 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.116761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:43.316374 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.316315 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:43.411259 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.411231 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:43.429931 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.429897 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff98467-4m269"] Jul 31 00:15:43.433490 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.433470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.436353 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.436329 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 31 00:15:43.436574 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.436412 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nl6x5\"" Jul 31 00:15:43.436574 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.436454 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 31 00:15:43.436574 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.436456 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 31 00:15:43.437051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.436726 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 31 00:15:43.452258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.452072 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff98467-4m269"] Jul 31 00:15:43.568668 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.568637 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:43.587811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.587771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.587811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.587812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zk84\" (UniqueName: \"kubernetes.io/projected/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-kube-api-access-5zk84\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.588037 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.587834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.588037 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.587926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-manifests\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.588037 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.587969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-tmp\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.688914 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.688821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.688914 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.688856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zk84\" (UniqueName: \"kubernetes.io/projected/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-kube-api-access-5zk84\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.688914 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.688881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.688914 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.688916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-manifests\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.689289 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.689041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-tmp\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.689362 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.689339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-manifests\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.691266 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.691239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-tmp\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.691510 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.691491 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.691510 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.691502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.702337 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.702311 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:43.707538 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.707509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zk84\" (UniqueName: \"kubernetes.io/projected/af0b51d3-ea77-4ded-b863-e3a2f62dcd3c-kube-api-access-5zk84\") pod \"opendatahub-operator-controller-manager-bff98467-4m269\" (UID: \"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c\") " pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.743631 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.743598 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:43.779177 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.779113 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:43.881303 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.881227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff98467-4m269"] Jul 31 00:15:43.884291 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:15:43.884249 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0b51d3_ea77_4ded_b863_e3a2f62dcd3c.slice/crio-70c93cdcd5fac8c8abf0a5e95c050bb1a8935d6db74901cb0b596da9ad89ba91 WatchSource:0}: Error finding container 70c93cdcd5fac8c8abf0a5e95c050bb1a8935d6db74901cb0b596da9ad89ba91: Status 404 returned error can't find the container with id 70c93cdcd5fac8c8abf0a5e95c050bb1a8935d6db74901cb0b596da9ad89ba91 Jul 31 00:15:43.884598 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.884575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:43.921825 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:43.921790 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" event={"ID":"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c","Type":"ContainerStarted","Data":"70c93cdcd5fac8c8abf0a5e95c050bb1a8935d6db74901cb0b596da9ad89ba91"} Jul 31 00:15:44.407580 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:44.407551 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:47.941881 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:47.941843 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" event={"ID":"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c","Type":"ContainerStarted","Data":"c9099361ec9109451b76ab50e32c091dcfd2541569d3f08b993b7699e5c20ca8"} Jul 31 00:15:48.762806 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:48.762747 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:15:48.946902 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:48.946869 2575 generic.go:356] "Generic (PLEG): container finished" podID="af0b51d3-ea77-4ded-b863-e3a2f62dcd3c" containerID="c9099361ec9109451b76ab50e32c091dcfd2541569d3f08b993b7699e5c20ca8" exitCode=0 Jul 31 00:15:48.947306 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:48.946955 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" event={"ID":"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c","Type":"ContainerDied","Data":"c9099361ec9109451b76ab50e32c091dcfd2541569d3f08b993b7699e5c20ca8"} Jul 31 00:15:49.953399 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:49.953354 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" event={"ID":"af0b51d3-ea77-4ded-b863-e3a2f62dcd3c","Type":"ContainerStarted","Data":"43ed3c28412459114844584e8c8e68cff0b11e16d0888c36f4b2f3619eb056df"} Jul 31 00:15:49.953874 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:49.953476 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:15:49.975176 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:49.975106 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" podStartSLOduration=2.3949660489999998 podStartE2EDuration="6.975084962s" totalImagesPullingTime="3.515800782s" totalInitContainerRuntime="1.064318131s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:15:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:15:49.972963075 +0000 UTC m=+559.495930820" watchObservedRunningTime="2026-07-31 00:15:49.975084962 +0000 UTC m=+559.498052766" Jul 31 00:15:52.133513 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:15:52.133479 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:00.959608 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:00.959571 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bff98467-4m269" Jul 31 00:16:03.352922 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:03.352887 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:03.478370 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:03.478336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:03.773489 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:03.773451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:03.928952 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:03.928919 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:04.028356 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.028278 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79"] Jul 31 00:16:04.034509 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.034479 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.037096 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.037041 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 31 00:16:04.037281 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.037260 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 00:16:04.038478 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.038456 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 31 00:16:04.038602 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.038510 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2qrj7\"" Jul 31 00:16:04.038602 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.038542 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 00:16:04.041698 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.041672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79"] Jul 31 00:16:04.096380 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.096342 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xqvts"] Jul 31 00:16:04.099198 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.099169 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:04.101902 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.101875 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lnnm7\"" Jul 31 00:16:04.102203 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.102184 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 31 00:16:04.110226 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.110189 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xqvts"] Jul 31 00:16:04.128038 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.128006 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:04.154398 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.154356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bab7c36f-d5a6-43ed-80ad-ceec181024ab-cert\") pod \"odh-model-controller-7998c67dc4-xqvts\" (UID: \"bab7c36f-d5a6-43ed-80ad-ceec181024ab\") " pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:04.154619 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.154421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32783c1-3abc-4735-9a8f-4bd705d05c20-tls-certs\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.154619 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.154469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzccc\" (UniqueName: \"kubernetes.io/projected/bab7c36f-d5a6-43ed-80ad-ceec181024ab-kube-api-access-nzccc\") pod \"odh-model-controller-7998c67dc4-xqvts\" (UID: \"bab7c36f-d5a6-43ed-80ad-ceec181024ab\") " pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:04.154619 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.154518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25c9\" (UniqueName: \"kubernetes.io/projected/a32783c1-3abc-4735-9a8f-4bd705d05c20-kube-api-access-p25c9\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.154619 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.154542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32783c1-3abc-4735-9a8f-4bd705d05c20-tmp\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.230818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.230781 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f6b47b856-hrcr4"] Jul 31 00:16:04.233068 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.233045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.238585 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.238558 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:04.252908 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.252870 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f6b47b856-hrcr4"] Jul 31 00:16:04.255598 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.255569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzccc\" (UniqueName: \"kubernetes.io/projected/bab7c36f-d5a6-43ed-80ad-ceec181024ab-kube-api-access-nzccc\") pod \"odh-model-controller-7998c67dc4-xqvts\" (UID: \"bab7c36f-d5a6-43ed-80ad-ceec181024ab\") " pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:04.255761 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.255639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25c9\" (UniqueName: \"kubernetes.io/projected/a32783c1-3abc-4735-9a8f-4bd705d05c20-kube-api-access-p25c9\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.255761 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.255674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32783c1-3abc-4735-9a8f-4bd705d05c20-tmp\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.255761 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.255729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bab7c36f-d5a6-43ed-80ad-ceec181024ab-cert\") pod \"odh-model-controller-7998c67dc4-xqvts\" (UID: \"bab7c36f-d5a6-43ed-80ad-ceec181024ab\") " pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:04.255922 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.255800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32783c1-3abc-4735-9a8f-4bd705d05c20-tls-certs\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.256168 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:04.256142 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 31 00:16:04.256412 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:04.256395 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bab7c36f-d5a6-43ed-80ad-ceec181024ab-cert podName:bab7c36f-d5a6-43ed-80ad-ceec181024ab nodeName:}" failed. No retries permitted until 2026-07-31 00:16:04.756361993 +0000 UTC m=+574.279329728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bab7c36f-d5a6-43ed-80ad-ceec181024ab-cert") pod "odh-model-controller-7998c67dc4-xqvts" (UID: "bab7c36f-d5a6-43ed-80ad-ceec181024ab") : secret "odh-model-controller-webhook-cert" not found Jul 31 00:16:04.258622 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.258595 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32783c1-3abc-4735-9a8f-4bd705d05c20-tmp\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.259043 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.259018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32783c1-3abc-4735-9a8f-4bd705d05c20-tls-certs\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.272784 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.272751 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzccc\" (UniqueName: \"kubernetes.io/projected/bab7c36f-d5a6-43ed-80ad-ceec181024ab-kube-api-access-nzccc\") pod \"odh-model-controller-7998c67dc4-xqvts\" (UID: \"bab7c36f-d5a6-43ed-80ad-ceec181024ab\") " pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:04.272921 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.272762 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25c9\" (UniqueName: \"kubernetes.io/projected/a32783c1-3abc-4735-9a8f-4bd705d05c20-kube-api-access-p25c9\") pod \"kube-auth-proxy-f6dcbc74-vjc79\" (UID: \"a32783c1-3abc-4735-9a8f-4bd705d05c20\") " pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.334483 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.334386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:04.345911 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.345874 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" Jul 31 00:16:04.356630 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.356588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-service-ca\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.357016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.356667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-oauth-serving-cert\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.357016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.356704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81610e93-bae2-4ce2-82b9-345daebeee94-console-oauth-config\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.357016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.356747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tthnn\" (UniqueName: \"kubernetes.io/projected/81610e93-bae2-4ce2-82b9-345daebeee94-kube-api-access-tthnn\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.357016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.356780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81610e93-bae2-4ce2-82b9-345daebeee94-console-serving-cert\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.357016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.356891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-trusted-ca-bundle\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.357016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.356962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-console-config\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.457842 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.457802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-trusted-ca-bundle\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.457976 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.457881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-console-config\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.457976 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.457915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-service-ca\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.457976 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.457948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-oauth-serving-cert\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.457976 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.457969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81610e93-bae2-4ce2-82b9-345daebeee94-console-oauth-config\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.458135 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.457997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tthnn\" (UniqueName: \"kubernetes.io/projected/81610e93-bae2-4ce2-82b9-345daebeee94-kube-api-access-tthnn\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.458135 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.458017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81610e93-bae2-4ce2-82b9-345daebeee94-console-serving-cert\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.459009 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.458977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-console-config\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.459181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.459071 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-service-ca\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.459421 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.459383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-oauth-serving-cert\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.459569 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.459526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81610e93-bae2-4ce2-82b9-345daebeee94-trusted-ca-bundle\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.461484 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.461436 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81610e93-bae2-4ce2-82b9-345daebeee94-console-serving-cert\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.462001 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.461973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81610e93-bae2-4ce2-82b9-345daebeee94-console-oauth-config\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.466528 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.466502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tthnn\" (UniqueName: \"kubernetes.io/projected/81610e93-bae2-4ce2-82b9-345daebeee94-kube-api-access-tthnn\") pod \"console-6f6b47b856-hrcr4\" (UID: \"81610e93-bae2-4ce2-82b9-345daebeee94\") " pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.490800 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.490774 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79"] Jul 31 00:16:04.493434 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:16:04.493405 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32783c1_3abc_4735_9a8f_4bd705d05c20.slice/crio-d8a3aab9ac17c15fd3293136b39f97cec08ec2a0e49c8c2fd1e268a0789968c6 WatchSource:0}: Error finding container d8a3aab9ac17c15fd3293136b39f97cec08ec2a0e49c8c2fd1e268a0789968c6: Status 404 returned error can't find the container with id d8a3aab9ac17c15fd3293136b39f97cec08ec2a0e49c8c2fd1e268a0789968c6 Jul 31 00:16:04.544323 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.544281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:04.714211 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.714178 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f6b47b856-hrcr4"] Jul 31 00:16:04.715940 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:16:04.715915 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81610e93_bae2_4ce2_82b9_345daebeee94.slice/crio-f79b7e84b15886f3d2db66f5f530828d19f1eaceba8a8699f30f0630c89e11ba WatchSource:0}: Error finding container f79b7e84b15886f3d2db66f5f530828d19f1eaceba8a8699f30f0630c89e11ba: Status 404 returned error can't find the container with id f79b7e84b15886f3d2db66f5f530828d19f1eaceba8a8699f30f0630c89e11ba Jul 31 00:16:04.761336 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.761303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bab7c36f-d5a6-43ed-80ad-ceec181024ab-cert\") pod \"odh-model-controller-7998c67dc4-xqvts\" (UID: \"bab7c36f-d5a6-43ed-80ad-ceec181024ab\") " pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:04.764169 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.764137 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bab7c36f-d5a6-43ed-80ad-ceec181024ab-cert\") pod \"odh-model-controller-7998c67dc4-xqvts\" (UID: \"bab7c36f-d5a6-43ed-80ad-ceec181024ab\") " pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:04.832986 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:04.832906 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:05.004995 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.004958 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vh2xp"] Jul 31 00:16:05.007667 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.007638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:05.009400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.009345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6b47b856-hrcr4" event={"ID":"81610e93-bae2-4ce2-82b9-345daebeee94","Type":"ContainerStarted","Data":"48abadd51bb3998eaa89d0998449cd39ebfefeaa4290dde4247ed5bd50e464c5"} Jul 31 00:16:05.009400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.009396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6b47b856-hrcr4" event={"ID":"81610e93-bae2-4ce2-82b9-345daebeee94","Type":"ContainerStarted","Data":"f79b7e84b15886f3d2db66f5f530828d19f1eaceba8a8699f30f0630c89e11ba"} Jul 31 00:16:05.010049 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.010023 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 31 00:16:05.010664 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.010641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" event={"ID":"a32783c1-3abc-4735-9a8f-4bd705d05c20","Type":"ContainerStarted","Data":"d8a3aab9ac17c15fd3293136b39f97cec08ec2a0e49c8c2fd1e268a0789968c6"} Jul 31 00:16:05.012062 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.012039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:05.018529 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.018501 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vh2xp"] Jul 31 00:16:05.030568 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.030320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:05.034329 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.034293 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z"] Jul 31 00:16:05.037158 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.037111 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.040241 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.039652 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 31 00:16:05.040241 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.039922 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g5nsg\"" Jul 31 00:16:05.040241 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.039928 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 31 00:16:05.040241 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.040104 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 31 00:16:05.040522 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.040301 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 31 00:16:05.052508 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.052296 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f6b47b856-hrcr4" podStartSLOduration=1.052274408 podStartE2EDuration="1.052274408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:16:05.051094637 +0000 UTC m=+574.574062383" watchObservedRunningTime="2026-07-31 00:16:05.052274408 +0000 UTC m=+574.575242154" Jul 31 00:16:05.052800 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.052738 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z"] Jul 31 00:16:05.067562 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.067526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfac874-6ea0-49f2-a296-6e2557d16606-tls-certs\") pod \"model-serving-api-54d89bd5fc-vh2xp\" (UID: \"8bfac874-6ea0-49f2-a296-6e2557d16606\") " pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:05.067828 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.067808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pps8q\" (UniqueName: \"kubernetes.io/projected/8bfac874-6ea0-49f2-a296-6e2557d16606-kube-api-access-pps8q\") pod \"model-serving-api-54d89bd5fc-vh2xp\" (UID: \"8bfac874-6ea0-49f2-a296-6e2557d16606\") " pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:05.123267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.123226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:05.169269 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfac874-6ea0-49f2-a296-6e2557d16606-tls-certs\") pod \"model-serving-api-54d89bd5fc-vh2xp\" (UID: \"8bfac874-6ea0-49f2-a296-6e2557d16606\") " pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:05.169463 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pps8q\" (UniqueName: \"kubernetes.io/projected/8bfac874-6ea0-49f2-a296-6e2557d16606-kube-api-access-pps8q\") pod \"model-serving-api-54d89bd5fc-vh2xp\" (UID: \"8bfac874-6ea0-49f2-a296-6e2557d16606\") " pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:05.169463 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.169463 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.169629 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.169629 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f310be-7555-4807-afec-11146e8ea879-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.169629 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:05.169354 2575 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 31 00:16:05.169629 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:05.169608 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfac874-6ea0-49f2-a296-6e2557d16606-tls-certs podName:8bfac874-6ea0-49f2-a296-6e2557d16606 nodeName:}" failed. No retries permitted until 2026-07-31 00:16:05.669586837 +0000 UTC m=+575.192554562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8bfac874-6ea0-49f2-a296-6e2557d16606-tls-certs") pod "model-serving-api-54d89bd5fc-vh2xp" (UID: "8bfac874-6ea0-49f2-a296-6e2557d16606") : secret "model-serving-api-tls" not found Jul 31 00:16:05.169840 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv98k\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-kube-api-access-bv98k\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.169840 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/98f310be-7555-4807-afec-11146e8ea879-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.169840 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.169743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.177071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.177050 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xqvts"] Jul 31 00:16:05.179634 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:16:05.179607 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab7c36f_d5a6_43ed_80ad_ceec181024ab.slice/crio-f483d60cdc14f313639447c52302935de7ef16ccad3473565b83e7d1751c34a4 WatchSource:0}: Error finding container f483d60cdc14f313639447c52302935de7ef16ccad3473565b83e7d1751c34a4: Status 404 returned error can't find the container with id f483d60cdc14f313639447c52302935de7ef16ccad3473565b83e7d1751c34a4 Jul 31 00:16:05.180092 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.180070 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pps8q\" (UniqueName: \"kubernetes.io/projected/8bfac874-6ea0-49f2-a296-6e2557d16606-kube-api-access-pps8q\") pod \"model-serving-api-54d89bd5fc-vh2xp\" (UID: \"8bfac874-6ea0-49f2-a296-6e2557d16606\") " pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:05.271402 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.271287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.271402 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.271365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.271402 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.271396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.271684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.271423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f310be-7555-4807-afec-11146e8ea879-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.271684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.271479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv98k\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-kube-api-access-bv98k\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.271684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.271510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/98f310be-7555-4807-afec-11146e8ea879-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.271684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.271539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.272330 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.272301 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/98f310be-7555-4807-afec-11146e8ea879-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.274436 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.274402 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f310be-7555-4807-afec-11146e8ea879-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.274945 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.274923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.275440 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.275379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.276064 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.276034 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.279354 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.279333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv98k\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-kube-api-access-bv98k\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.279905 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.279873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2zt6z\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.351575 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.351537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:05.483518 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.483457 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:05.531755 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.531718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z"] Jul 31 00:16:05.534328 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:16:05.534294 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f310be_7555_4807_afec_11146e8ea879.slice/crio-246de969507a9ccc3bbb5a37e8565f7823e07320627b05f855eb983b72ef0d6b WatchSource:0}: Error finding container 246de969507a9ccc3bbb5a37e8565f7823e07320627b05f855eb983b72ef0d6b: Status 404 returned error can't find the container with id 246de969507a9ccc3bbb5a37e8565f7823e07320627b05f855eb983b72ef0d6b Jul 31 00:16:05.676288 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.676249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfac874-6ea0-49f2-a296-6e2557d16606-tls-certs\") pod \"model-serving-api-54d89bd5fc-vh2xp\" (UID: \"8bfac874-6ea0-49f2-a296-6e2557d16606\") " pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:05.679037 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.679008 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfac874-6ea0-49f2-a296-6e2557d16606-tls-certs\") pod \"model-serving-api-54d89bd5fc-vh2xp\" (UID: \"8bfac874-6ea0-49f2-a296-6e2557d16606\") " pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:05.918170 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.918057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:05.921499 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:05.921466 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:06.016079 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:06.015983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" event={"ID":"98f310be-7555-4807-afec-11146e8ea879","Type":"ContainerStarted","Data":"246de969507a9ccc3bbb5a37e8565f7823e07320627b05f855eb983b72ef0d6b"} Jul 31 00:16:06.017355 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:06.017292 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" event={"ID":"bab7c36f-d5a6-43ed-80ad-ceec181024ab","Type":"ContainerStarted","Data":"f483d60cdc14f313639447c52302935de7ef16ccad3473565b83e7d1751c34a4"} Jul 31 00:16:06.110571 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:06.110521 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vh2xp"] Jul 31 00:16:06.114034 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:16:06.114002 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfac874_6ea0_49f2_a296_6e2557d16606.slice/crio-fc31217d165ba4b352a34dc58f52975775b7a1673d79f87fe7e0f6d149ec6c18 WatchSource:0}: Error finding container fc31217d165ba4b352a34dc58f52975775b7a1673d79f87fe7e0f6d149ec6c18: Status 404 returned error can't find the container with id fc31217d165ba4b352a34dc58f52975775b7a1673d79f87fe7e0f6d149ec6c18 Jul 31 00:16:07.023887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:07.023845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" event={"ID":"8bfac874-6ea0-49f2-a296-6e2557d16606","Type":"ContainerStarted","Data":"fc31217d165ba4b352a34dc58f52975775b7a1673d79f87fe7e0f6d149ec6c18"} Jul 31 00:16:10.386043 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:10.386001 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:16:10.386533 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:10.386084 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:16:11.066466 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.066209 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" event={"ID":"a32783c1-3abc-4735-9a8f-4bd705d05c20","Type":"ContainerStarted","Data":"37d5651acddd06536cabb7377ee839f836bbfb218ecdc6f29b12a77a9c9c8453"} Jul 31 00:16:11.066921 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.066895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" event={"ID":"8bfac874-6ea0-49f2-a296-6e2557d16606","Type":"ContainerStarted","Data":"5127d37fbb68fbc9d4de78a9abee9b5d464e5bd4d1ce20173a4d1eb129fd74f9"} Jul 31 00:16:11.067020 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.066988 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:11.069090 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.069062 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" event={"ID":"98f310be-7555-4807-afec-11146e8ea879","Type":"ContainerStarted","Data":"d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601"} Jul 31 00:16:11.070746 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.069650 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:11.071637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.071618 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2zt6z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 00:16:11.071788 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.071767 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" podUID="98f310be-7555-4807-afec-11146e8ea879" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 00:16:11.071862 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.071658 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" event={"ID":"bab7c36f-d5a6-43ed-80ad-ceec181024ab","Type":"ContainerStarted","Data":"282c53c3cb59fc12b828c487bca1b5b18f62112be2f5a1042d2b6be1da37aaa2"} Jul 31 00:16:11.072000 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.071947 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:11.082388 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.082075 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:11.082802 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.082752 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f6dcbc74-vjc79" podStartSLOduration=1.199228802 podStartE2EDuration="7.082733119s" totalImagesPullingTime="5.883504317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:16:11.080952478 +0000 UTC m=+580.603920268" watchObservedRunningTime="2026-07-31 00:16:11.082733119 +0000 UTC m=+580.605700869" Jul 31 00:16:11.115950 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.115890 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" podStartSLOduration=1.9055119 podStartE2EDuration="7.11587149s" totalImagesPullingTime="5.21035959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:16:11.113664193 +0000 UTC m=+580.636631960" watchObservedRunningTime="2026-07-31 00:16:11.11587149 +0000 UTC m=+580.638839235" Jul 31 00:16:11.145519 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.145458 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" podStartSLOduration=2.873179384 podStartE2EDuration="7.145440364s" totalImagesPullingTime="4.27226098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:16:11.143856963 +0000 UTC m=+580.666824734" watchObservedRunningTime="2026-07-31 00:16:11.145440364 +0000 UTC m=+580.668408110" Jul 31 00:16:11.174500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.174355 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" podStartSLOduration=1.326014407 podStartE2EDuration="6.174336026s" totalImagesPullingTime="4.848321619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:16:11.173791636 +0000 UTC m=+580.696759408" watchObservedRunningTime="2026-07-31 00:16:11.174336026 +0000 UTC m=+580.697303773" Jul 31 00:16:11.195746 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.195710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:11.430425 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.430387 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:11.463006 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.462967 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7"] Jul 31 00:16:11.466948 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.466903 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.469546 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.469523 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5wdn5\"" Jul 31 00:16:11.478291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.478254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7"] Jul 31 00:16:11.537106 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.537311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.537311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537196 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.537311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.537311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtcc\" (UniqueName: \"kubernetes.io/projected/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-kube-api-access-cxtcc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.537521 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.537521 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.537521 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.537521 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.537498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.633180 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.633144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:11.638320 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtcc\" (UniqueName: \"kubernetes.io/projected/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-kube-api-access-cxtcc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638708 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638708 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638708 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.638928 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.638902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.639234 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.639209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.639340 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.639234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.639340 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.639304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.639436 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.639337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.640980 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.640960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.641147 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.641110 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.646458 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.646429 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.647036 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.647009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtcc\" (UniqueName: \"kubernetes.io/projected/6d8018ac-2096-47a5-8bdd-af717ab2b8c6-kube-api-access-cxtcc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7\" (UID: \"6d8018ac-2096-47a5-8bdd-af717ab2b8c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.780964 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.780873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:11.798199 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.798169 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:11.923218 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:11.923191 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7"] Jul 31 00:16:11.926077 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:16:11.926047 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8018ac_2096_47a5_8bdd_af717ab2b8c6.slice/crio-87ff9e69eac87eb1dabc66bd33253d369d9ef5615fb34801d7069f4ad02bcac9 WatchSource:0}: Error finding container 87ff9e69eac87eb1dabc66bd33253d369d9ef5615fb34801d7069f4ad02bcac9: Status 404 returned error can't find the container with id 87ff9e69eac87eb1dabc66bd33253d369d9ef5615fb34801d7069f4ad02bcac9 Jul 31 00:16:12.076829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:12.076729 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" event={"ID":"6d8018ac-2096-47a5-8bdd-af717ab2b8c6","Type":"ContainerStarted","Data":"87ff9e69eac87eb1dabc66bd33253d369d9ef5615fb34801d7069f4ad02bcac9"} Jul 31 00:16:12.077633 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:12.077612 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:14.544886 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:14.544840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:14.545404 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:14.544997 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:14.551869 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:14.551819 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:15.027016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.026971 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:15.088161 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.088073 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-79ms7"] Jul 31 00:16:15.095842 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.095809 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t"] Jul 31 00:16:15.096422 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.096390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:15.101795 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.100190 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mftlv\"" Jul 31 00:16:15.101795 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.100443 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 31 00:16:15.106377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.106340 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f6b47b856-hrcr4" Jul 31 00:16:15.106678 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.106664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:15.108188 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.108163 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-79ms7"] Jul 31 00:16:15.109848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.109640 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 31 00:16:15.109848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.109657 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sm5hm\"" Jul 31 00:16:15.111638 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.111578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t"] Jul 31 00:16:15.127570 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.127538 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:15.168167 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.168133 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579f985bf4-jxjl4"] Jul 31 00:16:15.171438 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.171404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert\") pod \"kserve-controller-manager-d6f4b65b8-79ms7\" (UID: \"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:15.171609 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.171540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzd2\" (UniqueName: \"kubernetes.io/projected/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-kube-api-access-pdzd2\") pod \"kserve-controller-manager-d6f4b65b8-79ms7\" (UID: \"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:15.259569 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.259528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:15.272783 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.272740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f3ca15-7db5-4bd9-a057-ca804bfcfcce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tb58t\" (UID: \"17f3ca15-7db5-4bd9-a057-ca804bfcfcce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:15.272783 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.272788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzf9\" (UniqueName: \"kubernetes.io/projected/17f3ca15-7db5-4bd9-a057-ca804bfcfcce-kube-api-access-9kzf9\") pod \"llmisvc-controller-manager-777c6bd7d5-tb58t\" (UID: \"17f3ca15-7db5-4bd9-a057-ca804bfcfcce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:15.273029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.272834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert\") pod \"kserve-controller-manager-d6f4b65b8-79ms7\" (UID: \"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:15.273029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.272899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzd2\" (UniqueName: \"kubernetes.io/projected/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-kube-api-access-pdzd2\") pod \"kserve-controller-manager-d6f4b65b8-79ms7\" (UID: \"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:15.273029 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:15.273000 2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 31 00:16:15.273191 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:15.273071 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert podName:003e59a8-eb50-4c41-b4fb-ea86b9d87b5d nodeName:}" failed. No retries permitted until 2026-07-31 00:16:15.773049381 +0000 UTC m=+585.296017116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert") pod "kserve-controller-manager-d6f4b65b8-79ms7" (UID: "003e59a8-eb50-4c41-b4fb-ea86b9d87b5d") : secret "kserve-webhook-server-cert" not found Jul 31 00:16:15.293649 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.293556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzd2\" (UniqueName: \"kubernetes.io/projected/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-kube-api-access-pdzd2\") pod \"kserve-controller-manager-d6f4b65b8-79ms7\" (UID: \"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:15.374305 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.374259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f3ca15-7db5-4bd9-a057-ca804bfcfcce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tb58t\" (UID: \"17f3ca15-7db5-4bd9-a057-ca804bfcfcce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:15.374305 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.374311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzf9\" (UniqueName: \"kubernetes.io/projected/17f3ca15-7db5-4bd9-a057-ca804bfcfcce-kube-api-access-9kzf9\") pod \"llmisvc-controller-manager-777c6bd7d5-tb58t\" (UID: \"17f3ca15-7db5-4bd9-a057-ca804bfcfcce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:15.376603 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.376576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f3ca15-7db5-4bd9-a057-ca804bfcfcce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tb58t\" (UID: \"17f3ca15-7db5-4bd9-a057-ca804bfcfcce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:15.379153 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.379111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:15.382648 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.382615 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzf9\" (UniqueName: \"kubernetes.io/projected/17f3ca15-7db5-4bd9-a057-ca804bfcfcce-kube-api-access-9kzf9\") pod \"llmisvc-controller-manager-777c6bd7d5-tb58t\" (UID: \"17f3ca15-7db5-4bd9-a057-ca804bfcfcce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:15.408476 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.408439 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:16:15.408593 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.408536 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:16:15.408593 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.408582 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:16:15.431094 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.431060 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:15.517320 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.517287 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:15.589646 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.589558 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t"] Jul 31 00:16:15.591416 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.591366 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:15.593577 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:16:15.593553 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod17f3ca15_7db5_4bd9_a057_ca804bfcfcce.slice/crio-8367f3d440bd4b968e680a9b8b4385bc90c3eb785fbf5eb387a8d725dc419d3b WatchSource:0}: Error finding container 8367f3d440bd4b968e680a9b8b4385bc90c3eb785fbf5eb387a8d725dc419d3b: Status 404 returned error can't find the container with id 8367f3d440bd4b968e680a9b8b4385bc90c3eb785fbf5eb387a8d725dc419d3b Jul 31 00:16:15.778360 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.778327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert\") pod \"kserve-controller-manager-d6f4b65b8-79ms7\" (UID: \"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:15.778554 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:15.778452 2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 31 00:16:15.778554 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:15.778522 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert podName:003e59a8-eb50-4c41-b4fb-ea86b9d87b5d nodeName:}" failed. No retries permitted until 2026-07-31 00:16:16.77850781 +0000 UTC m=+586.301475532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert") pod "kserve-controller-manager-d6f4b65b8-79ms7" (UID: "003e59a8-eb50-4c41-b4fb-ea86b9d87b5d") : secret "kserve-webhook-server-cert" not found Jul 31 00:16:15.837951 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:15.837914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:16.102654 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.102615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" event={"ID":"17f3ca15-7db5-4bd9-a057-ca804bfcfcce","Type":"ContainerStarted","Data":"8367f3d440bd4b968e680a9b8b4385bc90c3eb785fbf5eb387a8d725dc419d3b"} Jul 31 00:16:16.104377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.104349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" event={"ID":"6d8018ac-2096-47a5-8bdd-af717ab2b8c6","Type":"ContainerStarted","Data":"d8243eb85131e7a459654899847773f00d540c0efaf48e083bff1d7f264b76f4"} Jul 31 00:16:16.144945 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.144880 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" podStartSLOduration=1.665597134 podStartE2EDuration="5.144861041s" totalImagesPullingTime="3.479263907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:16:16.143546806 +0000 UTC m=+585.666514586" watchObservedRunningTime="2026-07-31 00:16:16.144861041 +0000 UTC m=+585.667828786" Jul 31 00:16:16.297263 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.297230 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:16.398713 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.398619 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:16.496545 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.496510 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:16.781405 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.781366 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:16.787648 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.787615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert\") pod \"kserve-controller-manager-d6f4b65b8-79ms7\" (UID: \"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:16.787829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.787616 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:16.801561 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.800855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/003e59a8-eb50-4c41-b4fb-ea86b9d87b5d-cert\") pod \"kserve-controller-manager-d6f4b65b8-79ms7\" (UID: \"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:16.918318 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:16.918277 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:17.055569 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:17.055530 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-79ms7"] Jul 31 00:16:17.058049 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:16:17.058021 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003e59a8_eb50_4c41_b4fb_ea86b9d87b5d.slice/crio-3f94243a703d71919b3ec0ff623ffb3f2b32d087141bb4cbd06382db0853dc27 WatchSource:0}: Error finding container 3f94243a703d71919b3ec0ff623ffb3f2b32d087141bb4cbd06382db0853dc27: Status 404 returned error can't find the container with id 3f94243a703d71919b3ec0ff623ffb3f2b32d087141bb4cbd06382db0853dc27 Jul 31 00:16:17.112942 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:17.112908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" event={"ID":"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d","Type":"ContainerStarted","Data":"3f94243a703d71919b3ec0ff623ffb3f2b32d087141bb4cbd06382db0853dc27"} Jul 31 00:16:17.113318 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:17.113285 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:17.114329 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:17.114307 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7" Jul 31 00:16:21.135184 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.135145 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" event={"ID":"17f3ca15-7db5-4bd9-a057-ca804bfcfcce","Type":"ContainerStarted","Data":"9b1097e97202dfb24d4301556c3aed22d8b6e450d4116344117594a3fad36211"} Jul 31 00:16:21.135696 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.135208 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:21.136609 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.136584 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" event={"ID":"003e59a8-eb50-4c41-b4fb-ea86b9d87b5d","Type":"ContainerStarted","Data":"5dc1d773152cb4960caa5f6dbf9a32d10a8e66022217d4ab9fa306f063c4d752"} Jul 31 00:16:21.136742 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.136723 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:21.151197 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.151143 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" podStartSLOduration=1.722341514 podStartE2EDuration="6.151105218s" totalImagesPullingTime="4.428763704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:16:21.149527093 +0000 UTC m=+590.672494886" watchObservedRunningTime="2026-07-31 00:16:21.151105218 +0000 UTC m=+590.674072964" Jul 31 00:16:21.164671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.164621 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" podStartSLOduration=3.156412251 podStartE2EDuration="6.164605229s" totalImagesPullingTime="3.008192978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:16:21.163672417 +0000 UTC m=+590.686640191" watchObservedRunningTime="2026-07-31 00:16:21.164605229 +0000 UTC m=+590.687572973" Jul 31 00:16:21.681410 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.681375 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z"] Jul 31 00:16:21.681625 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.681603 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" podUID="98f310be-7555-4807-afec-11146e8ea879" containerName="discovery" containerID="cri-o://d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601" gracePeriod=30 Jul 31 00:16:21.937732 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:21.937666 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:22.036575 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.036540 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f310be-7555-4807-afec-11146e8ea879-local-certs\") pod \"98f310be-7555-4807-afec-11146e8ea879\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " Jul 31 00:16:22.036768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.036600 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-kubeconfig\") pod \"98f310be-7555-4807-afec-11146e8ea879\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " Jul 31 00:16:22.036768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.036619 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/98f310be-7555-4807-afec-11146e8ea879-istio-csr-ca-configmap\") pod \"98f310be-7555-4807-afec-11146e8ea879\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " Jul 31 00:16:22.036768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.036648 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-cacerts\") pod \"98f310be-7555-4807-afec-11146e8ea879\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " Jul 31 00:16:22.036768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.036664 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-istio-token\") pod \"98f310be-7555-4807-afec-11146e8ea879\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " Jul 31 00:16:22.036768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.036719 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-csr-dns-cert\") pod \"98f310be-7555-4807-afec-11146e8ea879\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " Jul 31 00:16:22.037030 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.036806 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv98k\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-kube-api-access-bv98k\") pod \"98f310be-7555-4807-afec-11146e8ea879\" (UID: \"98f310be-7555-4807-afec-11146e8ea879\") " Jul 31 00:16:22.037192 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.037162 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f310be-7555-4807-afec-11146e8ea879-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "98f310be-7555-4807-afec-11146e8ea879" (UID: "98f310be-7555-4807-afec-11146e8ea879"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:16:22.039427 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.039377 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-kube-api-access-bv98k" (OuterVolumeSpecName: "kube-api-access-bv98k") pod "98f310be-7555-4807-afec-11146e8ea879" (UID: "98f310be-7555-4807-afec-11146e8ea879"). InnerVolumeSpecName "kube-api-access-bv98k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:16:22.039555 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.039506 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-istio-token" (OuterVolumeSpecName: "istio-token") pod "98f310be-7555-4807-afec-11146e8ea879" (UID: "98f310be-7555-4807-afec-11146e8ea879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:16:22.039617 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.039594 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-cacerts" (OuterVolumeSpecName: "cacerts") pod "98f310be-7555-4807-afec-11146e8ea879" (UID: "98f310be-7555-4807-afec-11146e8ea879"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:16:22.039671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.039614 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "98f310be-7555-4807-afec-11146e8ea879" (UID: "98f310be-7555-4807-afec-11146e8ea879"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:16:22.039671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.039643 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "98f310be-7555-4807-afec-11146e8ea879" (UID: "98f310be-7555-4807-afec-11146e8ea879"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:16:22.039671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.039644 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f310be-7555-4807-afec-11146e8ea879-local-certs" (OuterVolumeSpecName: "local-certs") pod "98f310be-7555-4807-afec-11146e8ea879" (UID: "98f310be-7555-4807-afec-11146e8ea879"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:16:22.079632 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.079594 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-xqvts" Jul 31 00:16:22.082014 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.081996 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vh2xp" Jul 31 00:16:22.138169 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.138080 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv98k\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-kube-api-access-bv98k\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:22.138169 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.138130 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f310be-7555-4807-afec-11146e8ea879-local-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:22.138169 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.138148 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-kubeconfig\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:22.138169 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.138170 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/98f310be-7555-4807-afec-11146e8ea879-istio-csr-ca-configmap\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:22.138768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.138185 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-cacerts\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:22.138768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.138198 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f310be-7555-4807-afec-11146e8ea879-istio-token\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:22.138768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.138212 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/98f310be-7555-4807-afec-11146e8ea879-istio-csr-dns-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:22.148146 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.144516 2575 generic.go:356] "Generic (PLEG): container finished" podID="98f310be-7555-4807-afec-11146e8ea879" containerID="d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601" exitCode=0 Jul 31 00:16:22.148146 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.145212 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" Jul 31 00:16:22.149015 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.148460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" event={"ID":"98f310be-7555-4807-afec-11146e8ea879","Type":"ContainerDied","Data":"d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601"} Jul 31 00:16:22.149015 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.148525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z" event={"ID":"98f310be-7555-4807-afec-11146e8ea879","Type":"ContainerDied","Data":"246de969507a9ccc3bbb5a37e8565f7823e07320627b05f855eb983b72ef0d6b"} Jul 31 00:16:22.149015 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.148554 2575 scope.go:117] "RemoveContainer" containerID="d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601" Jul 31 00:16:22.167398 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.167370 2575 scope.go:117] "RemoveContainer" containerID="d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601" Jul 31 00:16:22.167765 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:22.167740 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601\": container with ID starting with d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601 not found: ID does not exist" containerID="d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601" Jul 31 00:16:22.167845 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.167776 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601"} err="failed to get container status \"d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601\": rpc error: code = NotFound desc = could not find container \"d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601\": container with ID starting with d80097ab8ad07d17d2d272cccb3d6b8ba43a638be78ca3bd2b5cec1c199ec601 not found: ID does not exist" Jul 31 00:16:22.175695 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.175665 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z"] Jul 31 00:16:22.185087 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:22.185054 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2zt6z"] Jul 31 00:16:23.048759 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:23.048729 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f310be-7555-4807-afec-11146e8ea879" path="/var/lib/kubelet/pods/98f310be-7555-4807-afec-11146e8ea879/volumes" Jul 31 00:16:33.164012 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:33.163974 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:33.616447 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:33.616362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:16:40.196421 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.196362 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-579f985bf4-jxjl4" podUID="1fd40b52-5468-4fc1-8c53-18a6ea26a963" containerName="console" containerID="cri-o://90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05" gracePeriod=15 Jul 31 00:16:40.442882 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.442849 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:16:40.501784 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.501690 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-oauth-serving-cert\") pod \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " Jul 31 00:16:40.501784 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.501736 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrrbp\" (UniqueName: \"kubernetes.io/projected/1fd40b52-5468-4fc1-8c53-18a6ea26a963-kube-api-access-wrrbp\") pod \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " Jul 31 00:16:40.501784 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.501767 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-oauth-config\") pod \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " Jul 31 00:16:40.502061 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.501805 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-serving-cert\") pod \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " Jul 31 00:16:40.502061 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.501828 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-service-ca\") pod \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " Jul 31 00:16:40.502061 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.501887 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-trusted-ca-bundle\") pod \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " Jul 31 00:16:40.502061 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.501913 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-config\") pod \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\" (UID: \"1fd40b52-5468-4fc1-8c53-18a6ea26a963\") " Jul 31 00:16:40.502264 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.502105 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1fd40b52-5468-4fc1-8c53-18a6ea26a963" (UID: "1fd40b52-5468-4fc1-8c53-18a6ea26a963"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:16:40.502400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.502367 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1fd40b52-5468-4fc1-8c53-18a6ea26a963" (UID: "1fd40b52-5468-4fc1-8c53-18a6ea26a963"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:16:40.502400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.502389 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-service-ca" (OuterVolumeSpecName: "service-ca") pod "1fd40b52-5468-4fc1-8c53-18a6ea26a963" (UID: "1fd40b52-5468-4fc1-8c53-18a6ea26a963"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:16:40.502400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.502393 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-config" (OuterVolumeSpecName: "console-config") pod "1fd40b52-5468-4fc1-8c53-18a6ea26a963" (UID: "1fd40b52-5468-4fc1-8c53-18a6ea26a963"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:16:40.504157 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.504098 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1fd40b52-5468-4fc1-8c53-18a6ea26a963" (UID: "1fd40b52-5468-4fc1-8c53-18a6ea26a963"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:16:40.504157 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.504146 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1fd40b52-5468-4fc1-8c53-18a6ea26a963" (UID: "1fd40b52-5468-4fc1-8c53-18a6ea26a963"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:16:40.504290 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.504247 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd40b52-5468-4fc1-8c53-18a6ea26a963-kube-api-access-wrrbp" (OuterVolumeSpecName: "kube-api-access-wrrbp") pod "1fd40b52-5468-4fc1-8c53-18a6ea26a963" (UID: "1fd40b52-5468-4fc1-8c53-18a6ea26a963"). InnerVolumeSpecName "kube-api-access-wrrbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:16:40.602839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.602797 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-trusted-ca-bundle\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:40.602839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.602829 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-config\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:40.602839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.602840 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-oauth-serving-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:40.602839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.602848 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrrbp\" (UniqueName: \"kubernetes.io/projected/1fd40b52-5468-4fc1-8c53-18a6ea26a963-kube-api-access-wrrbp\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:40.603110 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.602857 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-oauth-config\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:40.603110 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.602865 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fd40b52-5468-4fc1-8c53-18a6ea26a963-console-serving-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:40.603110 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:40.602874 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd40b52-5468-4fc1-8c53-18a6ea26a963-service-ca\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:16:41.231339 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.231306 2575 generic.go:356] "Generic (PLEG): container finished" podID="1fd40b52-5468-4fc1-8c53-18a6ea26a963" containerID="90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05" exitCode=0 Jul 31 00:16:41.231728 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.231350 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579f985bf4-jxjl4" Jul 31 00:16:41.231728 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.231356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579f985bf4-jxjl4" event={"ID":"1fd40b52-5468-4fc1-8c53-18a6ea26a963","Type":"ContainerDied","Data":"90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05"} Jul 31 00:16:41.231728 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.231397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579f985bf4-jxjl4" event={"ID":"1fd40b52-5468-4fc1-8c53-18a6ea26a963","Type":"ContainerDied","Data":"baf152bdfc297c914a27ad39dc223c14a9e29b642424100a722e4544892862a0"} Jul 31 00:16:41.231728 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.231417 2575 scope.go:117] "RemoveContainer" containerID="90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05" Jul 31 00:16:41.239584 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.239564 2575 scope.go:117] "RemoveContainer" containerID="90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05" Jul 31 00:16:41.239883 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:16:41.239863 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05\": container with ID starting with 90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05 not found: ID does not exist" containerID="90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05" Jul 31 00:16:41.239928 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.239893 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05"} err="failed to get container status \"90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05\": rpc error: code = NotFound desc = could not find container \"90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05\": container with ID starting with 90e00ed7e41e0a8ba64f55aebbbfb069e919c88b2bbeeb7ba541193760e51b05 not found: ID does not exist" Jul 31 00:16:41.247946 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.247921 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579f985bf4-jxjl4"] Jul 31 00:16:41.250064 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:41.250039 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-579f985bf4-jxjl4"] Jul 31 00:16:43.048458 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:43.048417 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fd40b52-5468-4fc1-8c53-18a6ea26a963" path="/var/lib/kubelet/pods/1fd40b52-5468-4fc1-8c53-18a6ea26a963/volumes" Jul 31 00:16:52.150364 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:52.150331 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-79ms7" Jul 31 00:16:52.151776 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:52.151749 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tb58t" Jul 31 00:16:55.191965 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:16:55.191921 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:05.158476 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:05.158429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:05.284049 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:05.284013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:18.667582 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:18.667549 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:18.721524 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:18.721497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:20.616111 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:20.616072 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:20.717906 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:20.717871 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:22.625312 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:22.625277 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:23.437168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:23.437137 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:25.905161 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.905113 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-28n2g"] Jul 31 00:17:25.905583 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.905450 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd40b52-5468-4fc1-8c53-18a6ea26a963" containerName="console" Jul 31 00:17:25.905583 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.905460 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd40b52-5468-4fc1-8c53-18a6ea26a963" containerName="console" Jul 31 00:17:25.905583 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.905469 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f310be-7555-4807-afec-11146e8ea879" containerName="discovery" Jul 31 00:17:25.905583 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.905475 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f310be-7555-4807-afec-11146e8ea879" containerName="discovery" Jul 31 00:17:25.905583 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.905532 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f310be-7555-4807-afec-11146e8ea879" containerName="discovery" Jul 31 00:17:25.905583 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.905540 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd40b52-5468-4fc1-8c53-18a6ea26a963" containerName="console" Jul 31 00:17:25.907377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.907358 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" Jul 31 00:17:25.910091 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.910062 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 00:17:25.910091 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.910066 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xgnhn\"" Jul 31 00:17:25.910304 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.910067 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 00:17:25.918475 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.918442 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-28n2g"] Jul 31 00:17:25.993866 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:25.993831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpfz\" (UniqueName: \"kubernetes.io/projected/242f44ec-cc86-4923-bf5a-70aa6a437839-kube-api-access-gwpfz\") pod \"authorino-operator-68d88948d8-28n2g\" (UID: \"242f44ec-cc86-4923-bf5a-70aa6a437839\") " pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" Jul 31 00:17:26.095045 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:26.095009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpfz\" (UniqueName: \"kubernetes.io/projected/242f44ec-cc86-4923-bf5a-70aa6a437839-kube-api-access-gwpfz\") pod \"authorino-operator-68d88948d8-28n2g\" (UID: \"242f44ec-cc86-4923-bf5a-70aa6a437839\") " pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" Jul 31 00:17:26.103968 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:26.103929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpfz\" (UniqueName: \"kubernetes.io/projected/242f44ec-cc86-4923-bf5a-70aa6a437839-kube-api-access-gwpfz\") pod \"authorino-operator-68d88948d8-28n2g\" (UID: \"242f44ec-cc86-4923-bf5a-70aa6a437839\") " pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" Jul 31 00:17:26.217942 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:26.217849 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" Jul 31 00:17:26.359321 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:26.359298 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-28n2g"] Jul 31 00:17:26.361909 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:17:26.361880 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242f44ec_cc86_4923_bf5a_70aa6a437839.slice/crio-b01cccf36a6b12f30cc415435b126e6291f0c1f385429bffb02be7207cd8d692 WatchSource:0}: Error finding container b01cccf36a6b12f30cc415435b126e6291f0c1f385429bffb02be7207cd8d692: Status 404 returned error can't find the container with id b01cccf36a6b12f30cc415435b126e6291f0c1f385429bffb02be7207cd8d692 Jul 31 00:17:26.427108 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:26.427073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" event={"ID":"242f44ec-cc86-4923-bf5a-70aa6a437839","Type":"ContainerStarted","Data":"b01cccf36a6b12f30cc415435b126e6291f0c1f385429bffb02be7207cd8d692"} Jul 31 00:17:26.634452 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:26.634359 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:26.874505 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:26.874461 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:28.216745 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.216709 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh"] Jul 31 00:17:28.223678 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.223650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:28.226365 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.226340 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tldwp\"" Jul 31 00:17:28.231273 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.231247 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh"] Jul 31 00:17:28.315711 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.315671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4422\" (UniqueName: \"kubernetes.io/projected/0678711d-802a-4ad4-9c12-e846fd589994-kube-api-access-v4422\") pod \"limitador-operator-controller-manager-6c549b6d85-58kjh\" (UID: \"0678711d-802a-4ad4-9c12-e846fd589994\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:28.416652 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.416609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4422\" (UniqueName: \"kubernetes.io/projected/0678711d-802a-4ad4-9c12-e846fd589994-kube-api-access-v4422\") pod \"limitador-operator-controller-manager-6c549b6d85-58kjh\" (UID: \"0678711d-802a-4ad4-9c12-e846fd589994\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:28.426237 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.426200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4422\" (UniqueName: \"kubernetes.io/projected/0678711d-802a-4ad4-9c12-e846fd589994-kube-api-access-v4422\") pod \"limitador-operator-controller-manager-6c549b6d85-58kjh\" (UID: \"0678711d-802a-4ad4-9c12-e846fd589994\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:28.538942 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.538852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:28.854666 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.854580 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:28.993633 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:28.993573 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh"] Jul 31 00:17:28.995805 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:17:28.995769 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0678711d_802a_4ad4_9c12_e846fd589994.slice/crio-78fb5d0e7bfa0127ad26103b7b874ae1e433085eb40ae5700e00a6b6de74d956 WatchSource:0}: Error finding container 78fb5d0e7bfa0127ad26103b7b874ae1e433085eb40ae5700e00a6b6de74d956: Status 404 returned error can't find the container with id 78fb5d0e7bfa0127ad26103b7b874ae1e433085eb40ae5700e00a6b6de74d956 Jul 31 00:17:29.134410 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:29.134320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:29.443689 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:29.443653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" event={"ID":"0678711d-802a-4ad4-9c12-e846fd589994","Type":"ContainerStarted","Data":"78fb5d0e7bfa0127ad26103b7b874ae1e433085eb40ae5700e00a6b6de74d956"} Jul 31 00:17:29.445311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:29.445281 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" event={"ID":"242f44ec-cc86-4923-bf5a-70aa6a437839","Type":"ContainerStarted","Data":"f37646adba526659a698b33eb4a070f78d3d33ec1af3f6ad71dd7a53dc05d139"} Jul 31 00:17:29.445427 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:29.445353 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" Jul 31 00:17:29.462385 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:29.462338 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" podStartSLOduration=1.9303416580000001 podStartE2EDuration="4.462321311s" totalImagesPullingTime="2.531979653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:17:29.460901634 +0000 UTC m=+658.983869408" watchObservedRunningTime="2026-07-31 00:17:29.462321311 +0000 UTC m=+658.985289055" Jul 31 00:17:30.461901 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.461862 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf"] Jul 31 00:17:30.464362 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.464343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" Jul 31 00:17:30.466916 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.466895 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 31 00:17:30.467041 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.466895 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-snh9d\"" Jul 31 00:17:30.475956 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.475895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf"] Jul 31 00:17:30.533087 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.533042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshrw\" (UniqueName: \"kubernetes.io/projected/b488704f-6e0c-434c-8fa5-22500a86c020-kube-api-access-lshrw\") pod \"dns-operator-controller-manager-5985564776-qxpsf\" (UID: \"b488704f-6e0c-434c-8fa5-22500a86c020\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" Jul 31 00:17:30.634435 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.634387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshrw\" (UniqueName: \"kubernetes.io/projected/b488704f-6e0c-434c-8fa5-22500a86c020-kube-api-access-lshrw\") pod \"dns-operator-controller-manager-5985564776-qxpsf\" (UID: \"b488704f-6e0c-434c-8fa5-22500a86c020\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" Jul 31 00:17:30.644479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.644446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshrw\" (UniqueName: \"kubernetes.io/projected/b488704f-6e0c-434c-8fa5-22500a86c020-kube-api-access-lshrw\") pod \"dns-operator-controller-manager-5985564776-qxpsf\" (UID: \"b488704f-6e0c-434c-8fa5-22500a86c020\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" Jul 31 00:17:30.778045 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.777956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" Jul 31 00:17:30.859465 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:30.859435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:31.081258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:31.081233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf"] Jul 31 00:17:31.083606 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:17:31.083574 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb488704f_6e0c_434c_8fa5_22500a86c020.slice/crio-2f308847cd5de1cae240182a14eab2fb427c4054cc9395094d4b9dd0fe018e4b WatchSource:0}: Error finding container 2f308847cd5de1cae240182a14eab2fb427c4054cc9395094d4b9dd0fe018e4b: Status 404 returned error can't find the container with id 2f308847cd5de1cae240182a14eab2fb427c4054cc9395094d4b9dd0fe018e4b Jul 31 00:17:31.225530 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:31.225485 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:31.455062 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:31.455021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" event={"ID":"b488704f-6e0c-434c-8fa5-22500a86c020","Type":"ContainerStarted","Data":"2f308847cd5de1cae240182a14eab2fb427c4054cc9395094d4b9dd0fe018e4b"} Jul 31 00:17:31.456701 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:31.456678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" event={"ID":"0678711d-802a-4ad4-9c12-e846fd589994","Type":"ContainerStarted","Data":"9f560f3f7b63cb8c4f76c9a168842bf2d499116900aa866e7a0e887e169f15bb"} Jul 31 00:17:31.456832 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:31.456779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:31.468378 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:31.468351 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:31.478425 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:31.478375 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" podStartSLOduration=1.488756309 podStartE2EDuration="3.478360518s" totalImagesPullingTime="1.989604209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:17:31.476543804 +0000 UTC m=+660.999511550" watchObservedRunningTime="2026-07-31 00:17:31.478360518 +0000 UTC m=+661.001328262" Jul 31 00:17:31.639164 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:31.639135 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:32.887798 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:32.887757 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:32.930214 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:32.930190 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 31 00:17:33.467052 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:33.467015 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" event={"ID":"b488704f-6e0c-434c-8fa5-22500a86c020","Type":"ContainerStarted","Data":"047c7ee3a8e654310635099138055b8e2b0c89ecb3c78b8e53ce6b346d8fac17"} Jul 31 00:17:33.467257 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:33.467095 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" Jul 31 00:17:33.489016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:33.488955 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" podStartSLOduration=1.6471414009999998 podStartE2EDuration="3.488937912s" totalImagesPullingTime="1.841796511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:17:33.488761219 +0000 UTC m=+663.011728989" watchObservedRunningTime="2026-07-31 00:17:33.488937912 +0000 UTC m=+663.011905657" Jul 31 00:17:34.483295 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.483266 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl"] Jul 31 00:17:34.485697 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.485674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:34.488276 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.488254 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r29qw\"" Jul 31 00:17:34.496845 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.496819 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl"] Jul 31 00:17:34.568711 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.568679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49e7d3de-d98d-4fbf-9140-4728cb94dc00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" (UID: \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:34.568895 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.568731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvgv\" (UniqueName: \"kubernetes.io/projected/49e7d3de-d98d-4fbf-9140-4728cb94dc00-kube-api-access-mjvgv\") pod \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" (UID: \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:34.669561 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.669512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49e7d3de-d98d-4fbf-9140-4728cb94dc00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" (UID: \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:34.669561 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.669566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvgv\" (UniqueName: \"kubernetes.io/projected/49e7d3de-d98d-4fbf-9140-4728cb94dc00-kube-api-access-mjvgv\") pod \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" (UID: \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:34.670018 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.669990 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49e7d3de-d98d-4fbf-9140-4728cb94dc00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" (UID: \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:34.677265 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.677244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvgv\" (UniqueName: \"kubernetes.io/projected/49e7d3de-d98d-4fbf-9140-4728cb94dc00-kube-api-access-mjvgv\") pod \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" (UID: \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:34.797063 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.796967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:34.951133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:34.951079 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl"] Jul 31 00:17:34.954653 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:17:34.954626 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e7d3de_d98d_4fbf_9140_4728cb94dc00.slice/crio-c1c1f01912ce3cad1de9303d7b6608d25bcdd0a75a72f47e8bcb461bcf053a3c WatchSource:0}: Error finding container c1c1f01912ce3cad1de9303d7b6608d25bcdd0a75a72f47e8bcb461bcf053a3c: Status 404 returned error can't find the container with id c1c1f01912ce3cad1de9303d7b6608d25bcdd0a75a72f47e8bcb461bcf053a3c Jul 31 00:17:35.477636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:35.477598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" event={"ID":"49e7d3de-d98d-4fbf-9140-4728cb94dc00","Type":"ContainerStarted","Data":"c1c1f01912ce3cad1de9303d7b6608d25bcdd0a75a72f47e8bcb461bcf053a3c"} Jul 31 00:17:36.445319 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:36.445282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:36.542629 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:36.542591 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:37.241907 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:37.241871 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:37.341035 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:37.340998 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:39.501026 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:39.500932 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" event={"ID":"49e7d3de-d98d-4fbf-9140-4728cb94dc00","Type":"ContainerStarted","Data":"5573fceb686f35eb930aefa9507dd76d053e57a1bc6e9604ea799705b8a4466b"} Jul 31 00:17:39.501391 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:39.501052 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:39.524076 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:39.524022 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" podStartSLOduration=1.3021798549999999 podStartE2EDuration="5.524002606s" totalImagesPullingTime="4.221822751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:17:39.521330364 +0000 UTC m=+669.044298129" watchObservedRunningTime="2026-07-31 00:17:39.524002606 +0000 UTC m=+669.046970350" Jul 31 00:17:40.452267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:40.452236 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-28n2g" Jul 31 00:17:40.636460 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:40.636426 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:40.739356 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:40.739273 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:41.877954 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.877916 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:41.878765 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.878742 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv"] Jul 31 00:17:41.881194 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.881177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:41.883624 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.883601 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 31 00:17:41.883764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.883644 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 31 00:17:41.883764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.883647 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5248n\"" Jul 31 00:17:41.889332 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.889305 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv"] Jul 31 00:17:41.935194 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.935143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvd2g\" (UniqueName: \"kubernetes.io/projected/54239473-e042-4670-a5b7-ca4cdcb1ad20-kube-api-access-xvd2g\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:41.935381 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.935232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/54239473-e042-4670-a5b7-ca4cdcb1ad20-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:41.935381 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:41.935277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54239473-e042-4670-a5b7-ca4cdcb1ad20-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.036807 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.036770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvd2g\" (UniqueName: \"kubernetes.io/projected/54239473-e042-4670-a5b7-ca4cdcb1ad20-kube-api-access-xvd2g\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.037096 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.037064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/54239473-e042-4670-a5b7-ca4cdcb1ad20-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.037222 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.037158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54239473-e042-4670-a5b7-ca4cdcb1ad20-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.037222 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:42.037174 2575 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 31 00:17:42.037308 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:42.037237 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54239473-e042-4670-a5b7-ca4cdcb1ad20-plugin-serving-cert podName:54239473-e042-4670-a5b7-ca4cdcb1ad20 nodeName:}" failed. No retries permitted until 2026-07-31 00:17:42.537217831 +0000 UTC m=+672.060185558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/54239473-e042-4670-a5b7-ca4cdcb1ad20-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-kk5lv" (UID: "54239473-e042-4670-a5b7-ca4cdcb1ad20") : secret "plugin-serving-cert" not found Jul 31 00:17:42.037732 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.037715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54239473-e042-4670-a5b7-ca4cdcb1ad20-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.045060 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.045026 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvd2g\" (UniqueName: \"kubernetes.io/projected/54239473-e042-4670-a5b7-ca4cdcb1ad20-kube-api-access-xvd2g\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.147781 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.147688 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:42.248251 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.248216 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:42.349384 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.349344 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:42.463684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.463652 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:42.542832 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.542789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/54239473-e042-4670-a5b7-ca4cdcb1ad20-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.545461 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.545433 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/54239473-e042-4670-a5b7-ca4cdcb1ad20-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kk5lv\" (UID: \"54239473-e042-4670-a5b7-ca4cdcb1ad20\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.790787 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.790698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" Jul 31 00:17:42.928716 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:42.928685 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv"] Jul 31 00:17:42.930617 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:17:42.930589 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54239473_e042_4670_a5b7_ca4cdcb1ad20.slice/crio-29a4c44b904f1c5352c55aaa199f8b37afc5fe15b5f062e55d74171070b23147 WatchSource:0}: Error finding container 29a4c44b904f1c5352c55aaa199f8b37afc5fe15b5f062e55d74171070b23147: Status 404 returned error can't find the container with id 29a4c44b904f1c5352c55aaa199f8b37afc5fe15b5f062e55d74171070b23147 Jul 31 00:17:43.520990 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:43.520954 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" event={"ID":"54239473-e042-4670-a5b7-ca4cdcb1ad20","Type":"ContainerStarted","Data":"29a4c44b904f1c5352c55aaa199f8b37afc5fe15b5f062e55d74171070b23147"} Jul 31 00:17:44.474256 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:44.474219 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-qxpsf" Jul 31 00:17:48.550529 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:48.550482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" event={"ID":"54239473-e042-4670-a5b7-ca4cdcb1ad20","Type":"ContainerStarted","Data":"bfccf3d1142ffa7e6e5e86a3bb61b53f72a9ed0b0bc8bd8c294273438b1f0b6a"} Jul 31 00:17:48.598827 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:48.598765 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kk5lv" podStartSLOduration=2.850695168 podStartE2EDuration="7.598741545s" totalImagesPullingTime="4.748046377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:17:48.597863129 +0000 UTC m=+678.120830875" watchObservedRunningTime="2026-07-31 00:17:48.598741545 +0000 UTC m=+678.121709292" Jul 31 00:17:48.627735 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:48.627692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:48.708113 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:48.708074 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:48.779079 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:48.779041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:50.507751 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:50.507715 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:52.277783 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.277746 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl"] Jul 31 00:17:52.278255 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.277920 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" containerName="manager" containerID="cri-o://5573fceb686f35eb930aefa9507dd76d053e57a1bc6e9604ea799705b8a4466b" gracePeriod=2 Jul 31 00:17:52.283292 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.283255 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl"] Jul 31 00:17:52.299008 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.298982 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn"] Jul 31 00:17:52.299379 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.299364 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" containerName="manager" Jul 31 00:17:52.299379 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.299378 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" containerName="manager" Jul 31 00:17:52.299500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.299451 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" containerName="manager" Jul 31 00:17:52.301475 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.301454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:52.303812 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:52.303781 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:52.314061 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.314017 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh"] Jul 31 00:17:52.314401 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.314321 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" podUID="0678711d-802a-4ad4-9c12-e846fd589994" containerName="manager" containerID="cri-o://9f560f3f7b63cb8c4f76c9a168842bf2d499116900aa866e7a0e887e169f15bb" gracePeriod=2 Jul 31 00:17:52.317166 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.317142 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn"] Jul 31 00:17:52.341898 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:52.341866 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:52.416957 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.416914 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh"] Jul 31 00:17:52.429061 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.428984 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5"] Jul 31 00:17:52.429538 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.429504 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0678711d-802a-4ad4-9c12-e846fd589994" containerName="manager" Jul 31 00:17:52.429538 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.429531 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0678711d-802a-4ad4-9c12-e846fd589994" containerName="manager" Jul 31 00:17:52.429691 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.429624 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0678711d-802a-4ad4-9c12-e846fd589994" containerName="manager" Jul 31 00:17:52.431810 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.431790 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" Jul 31 00:17:52.434425 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.434399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/508bfa7d-b73b-4c90-94d6-a824e80db2a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-775tn\" (UID: \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:52.434526 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.434447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnxm\" (UniqueName: \"kubernetes.io/projected/508bfa7d-b73b-4c90-94d6-a824e80db2a0-kube-api-access-ppnxm\") pod \"kuadrant-operator-controller-manager-58fc56757c-775tn\" (UID: \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:52.445734 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.442534 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5"] Jul 31 00:17:52.458725 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:52.458690 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:52.460667 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:52.460636 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-58kjh\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="0678711d-802a-4ad4-9c12-e846fd589994" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:52.535030 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.534936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/508bfa7d-b73b-4c90-94d6-a824e80db2a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-775tn\" (UID: \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:52.535030 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.535015 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnxm\" (UniqueName: \"kubernetes.io/projected/508bfa7d-b73b-4c90-94d6-a824e80db2a0-kube-api-access-ppnxm\") pod \"kuadrant-operator-controller-manager-58fc56757c-775tn\" (UID: \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:52.535262 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.535064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrlc\" (UniqueName: \"kubernetes.io/projected/d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe-kube-api-access-qkrlc\") pod \"limitador-operator-controller-manager-6c549b6d85-dkxk5\" (UID: \"d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" Jul 31 00:17:52.535404 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.535381 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/508bfa7d-b73b-4c90-94d6-a824e80db2a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-775tn\" (UID: \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:52.549027 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.549006 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnxm\" (UniqueName: \"kubernetes.io/projected/508bfa7d-b73b-4c90-94d6-a824e80db2a0-kube-api-access-ppnxm\") pod \"kuadrant-operator-controller-manager-58fc56757c-775tn\" (UID: \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:52.570301 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.570271 2575 generic.go:356] "Generic (PLEG): container finished" podID="0678711d-802a-4ad4-9c12-e846fd589994" containerID="9f560f3f7b63cb8c4f76c9a168842bf2d499116900aa866e7a0e887e169f15bb" exitCode=0 Jul 31 00:17:52.570431 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.570360 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78fb5d0e7bfa0127ad26103b7b874ae1e433085eb40ae5700e00a6b6de74d956" Jul 31 00:17:52.571782 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.571759 2575 generic.go:356] "Generic (PLEG): container finished" podID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" containerID="5573fceb686f35eb930aefa9507dd76d053e57a1bc6e9604ea799705b8a4466b" exitCode=0 Jul 31 00:17:52.571891 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.571834 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1c1f01912ce3cad1de9303d7b6608d25bcdd0a75a72f47e8bcb461bcf053a3c" Jul 31 00:17:52.577109 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.577087 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:52.579470 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:52.579440 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:52.579470 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.579463 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:52.581672 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:52.581641 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-58kjh\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="0678711d-802a-4ad4-9c12-e846fd589994" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:52.583560 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:52.583537 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nnxhl\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:52.585394 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:17:52.585371 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-58kjh\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="0678711d-802a-4ad4-9c12-e846fd589994" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:52.635714 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.635678 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjvgv\" (UniqueName: \"kubernetes.io/projected/49e7d3de-d98d-4fbf-9140-4728cb94dc00-kube-api-access-mjvgv\") pod \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\" (UID: \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\") " Jul 31 00:17:52.635902 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.635750 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49e7d3de-d98d-4fbf-9140-4728cb94dc00-extensions-socket-volume\") pod \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\" (UID: \"49e7d3de-d98d-4fbf-9140-4728cb94dc00\") " Jul 31 00:17:52.635961 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.635911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrlc\" (UniqueName: \"kubernetes.io/projected/d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe-kube-api-access-qkrlc\") pod \"limitador-operator-controller-manager-6c549b6d85-dkxk5\" (UID: \"d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" Jul 31 00:17:52.636257 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.636232 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e7d3de-d98d-4fbf-9140-4728cb94dc00-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "49e7d3de-d98d-4fbf-9140-4728cb94dc00" (UID: "49e7d3de-d98d-4fbf-9140-4728cb94dc00"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:17:52.637913 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.637892 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e7d3de-d98d-4fbf-9140-4728cb94dc00-kube-api-access-mjvgv" (OuterVolumeSpecName: "kube-api-access-mjvgv") pod "49e7d3de-d98d-4fbf-9140-4728cb94dc00" (UID: "49e7d3de-d98d-4fbf-9140-4728cb94dc00"). InnerVolumeSpecName "kube-api-access-mjvgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:17:52.649977 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.649940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrlc\" (UniqueName: \"kubernetes.io/projected/d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe-kube-api-access-qkrlc\") pod \"limitador-operator-controller-manager-6c549b6d85-dkxk5\" (UID: \"d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" Jul 31 00:17:52.706427 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.706395 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:52.736626 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.736590 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4422\" (UniqueName: \"kubernetes.io/projected/0678711d-802a-4ad4-9c12-e846fd589994-kube-api-access-v4422\") pod \"0678711d-802a-4ad4-9c12-e846fd589994\" (UID: \"0678711d-802a-4ad4-9c12-e846fd589994\") " Jul 31 00:17:52.736807 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.736765 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/49e7d3de-d98d-4fbf-9140-4728cb94dc00-extensions-socket-volume\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:17:52.736807 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.736781 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvgv\" (UniqueName: \"kubernetes.io/projected/49e7d3de-d98d-4fbf-9140-4728cb94dc00-kube-api-access-mjvgv\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:17:52.738674 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.738643 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0678711d-802a-4ad4-9c12-e846fd589994-kube-api-access-v4422" (OuterVolumeSpecName: "kube-api-access-v4422") pod "0678711d-802a-4ad4-9c12-e846fd589994" (UID: "0678711d-802a-4ad4-9c12-e846fd589994"). InnerVolumeSpecName "kube-api-access-v4422". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:17:52.750785 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.750753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" Jul 31 00:17:52.837885 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.837853 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4422\" (UniqueName: \"kubernetes.io/projected/0678711d-802a-4ad4-9c12-e846fd589994-kube-api-access-v4422\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:17:52.856306 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.856277 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn"] Jul 31 00:17:52.857988 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:17:52.857955 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508bfa7d_b73b_4c90_94d6_a824e80db2a0.slice/crio-07346710a7003c6f4bd4c8e75be8ff2e1ec0ed41e82e0e097508603839d5df5c WatchSource:0}: Error finding container 07346710a7003c6f4bd4c8e75be8ff2e1ec0ed41e82e0e097508603839d5df5c: Status 404 returned error can't find the container with id 07346710a7003c6f4bd4c8e75be8ff2e1ec0ed41e82e0e097508603839d5df5c Jul 31 00:17:52.900749 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:52.900722 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5"] Jul 31 00:17:52.914255 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:17:52.914220 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53807b2_f7a6_4bd0_a7cd_b6ddc019bdfe.slice/crio-c5deb44cd279ccd9e5db3d6e545ce24b906f54411184b4833ae18f0055f0bf5f WatchSource:0}: Error finding container c5deb44cd279ccd9e5db3d6e545ce24b906f54411184b4833ae18f0055f0bf5f: Status 404 returned error can't find the container with id c5deb44cd279ccd9e5db3d6e545ce24b906f54411184b4833ae18f0055f0bf5f Jul 31 00:17:53.048624 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.048544 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0678711d-802a-4ad4-9c12-e846fd589994" path="/var/lib/kubelet/pods/0678711d-802a-4ad4-9c12-e846fd589994/volumes" Jul 31 00:17:53.048948 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.048934 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49e7d3de-d98d-4fbf-9140-4728cb94dc00" path="/var/lib/kubelet/pods/49e7d3de-d98d-4fbf-9140-4728cb94dc00/volumes" Jul 31 00:17:53.462050 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.461977 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" podUID="0678711d-802a-4ad4-9c12-e846fd589994" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 31 00:17:53.578168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.578057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" event={"ID":"d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe","Type":"ContainerStarted","Data":"7d3b5c01bd3bb1bcf9428294217c33ebd98cebbf12773b9fce40859c6eb5c9d2"} Jul 31 00:17:53.578168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.578160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" event={"ID":"d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe","Type":"ContainerStarted","Data":"c5deb44cd279ccd9e5db3d6e545ce24b906f54411184b4833ae18f0055f0bf5f"} Jul 31 00:17:53.578394 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.578259 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" Jul 31 00:17:53.579848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.579812 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" event={"ID":"508bfa7d-b73b-4c90-94d6-a824e80db2a0","Type":"ContainerStarted","Data":"6d80103403205bbf53257663ebdd8ce3e6fa10d213036f8580226ad39888c684"} Jul 31 00:17:53.579848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.579843 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-58kjh" Jul 31 00:17:53.580047 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.579860 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" event={"ID":"508bfa7d-b73b-4c90-94d6-a824e80db2a0","Type":"ContainerStarted","Data":"07346710a7003c6f4bd4c8e75be8ff2e1ec0ed41e82e0e097508603839d5df5c"} Jul 31 00:17:53.580047 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.579827 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nnxhl" Jul 31 00:17:53.580047 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.579979 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:17:53.597547 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.597488 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" podStartSLOduration=1.597471608 podStartE2EDuration="1.597471608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:17:53.596001017 +0000 UTC m=+683.118968783" watchObservedRunningTime="2026-07-31 00:17:53.597471608 +0000 UTC m=+683.120439354" Jul 31 00:17:53.614262 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:53.614209 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" podStartSLOduration=1.614190885 podStartE2EDuration="1.614190885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:17:53.612873734 +0000 UTC m=+683.135841496" watchObservedRunningTime="2026-07-31 00:17:53.614190885 +0000 UTC m=+683.137158631" Jul 31 00:17:56.608643 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:56.608608 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:56.658729 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:56.658688 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:57.209254 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:57.209216 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:17:57.310309 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:17:57.310279 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:00.416349 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:00.416309 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:00.517762 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:00.517699 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:02.807621 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:02.807587 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:02.852861 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:02.852829 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:04.588321 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:04.588290 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-dkxk5" Jul 31 00:18:04.588777 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:04.588375 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:18:08.554912 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:08.554874 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:08.656720 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:08.656684 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:08.755819 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:08.755787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:12.482389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.482347 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn"] Jul 31 00:18:12.482858 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.482632 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" podUID="508bfa7d-b73b-4c90-94d6-a824e80db2a0" containerName="manager" containerID="cri-o://6d80103403205bbf53257663ebdd8ce3e6fa10d213036f8580226ad39888c684" gracePeriod=10 Jul 31 00:18:12.551721 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:18:12.551683 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508bfa7d_b73b_4c90_94d6_a824e80db2a0.slice/crio-conmon-6d80103403205bbf53257663ebdd8ce3e6fa10d213036f8580226ad39888c684.scope\": RecentStats: unable to find data in memory cache]" Jul 31 00:18:12.677596 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.677559 2575 generic.go:356] "Generic (PLEG): container finished" podID="508bfa7d-b73b-4c90-94d6-a824e80db2a0" containerID="6d80103403205bbf53257663ebdd8ce3e6fa10d213036f8580226ad39888c684" exitCode=0 Jul 31 00:18:12.677781 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.677627 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" event={"ID":"508bfa7d-b73b-4c90-94d6-a824e80db2a0","Type":"ContainerDied","Data":"6d80103403205bbf53257663ebdd8ce3e6fa10d213036f8580226ad39888c684"} Jul 31 00:18:12.711461 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.711427 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd"] Jul 31 00:18:12.714288 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.714270 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:12.727949 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.727918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd"] Jul 31 00:18:12.729640 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.729611 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:18:12.811798 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.811759 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/508bfa7d-b73b-4c90-94d6-a824e80db2a0-extensions-socket-volume\") pod \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\" (UID: \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\") " Jul 31 00:18:12.811985 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.811805 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppnxm\" (UniqueName: \"kubernetes.io/projected/508bfa7d-b73b-4c90-94d6-a824e80db2a0-kube-api-access-ppnxm\") pod \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\" (UID: \"508bfa7d-b73b-4c90-94d6-a824e80db2a0\") " Jul 31 00:18:12.811985 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.811928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8186d25-2a7e-4117-bd1a-5023ed575ec1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-56xpd\" (UID: \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:12.812088 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.812058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7z2\" (UniqueName: \"kubernetes.io/projected/e8186d25-2a7e-4117-bd1a-5023ed575ec1-kube-api-access-hd7z2\") pod \"kuadrant-operator-controller-manager-68cf589589-56xpd\" (UID: \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:12.812177 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.812156 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508bfa7d-b73b-4c90-94d6-a824e80db2a0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "508bfa7d-b73b-4c90-94d6-a824e80db2a0" (UID: "508bfa7d-b73b-4c90-94d6-a824e80db2a0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:18:12.813848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.813817 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508bfa7d-b73b-4c90-94d6-a824e80db2a0-kube-api-access-ppnxm" (OuterVolumeSpecName: "kube-api-access-ppnxm") pod "508bfa7d-b73b-4c90-94d6-a824e80db2a0" (UID: "508bfa7d-b73b-4c90-94d6-a824e80db2a0"). InnerVolumeSpecName "kube-api-access-ppnxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:18:12.912551 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.912503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7z2\" (UniqueName: \"kubernetes.io/projected/e8186d25-2a7e-4117-bd1a-5023ed575ec1-kube-api-access-hd7z2\") pod \"kuadrant-operator-controller-manager-68cf589589-56xpd\" (UID: \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:12.912746 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.912569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8186d25-2a7e-4117-bd1a-5023ed575ec1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-56xpd\" (UID: \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:12.912746 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.912632 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/508bfa7d-b73b-4c90-94d6-a824e80db2a0-extensions-socket-volume\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:18:12.912746 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.912647 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppnxm\" (UniqueName: \"kubernetes.io/projected/508bfa7d-b73b-4c90-94d6-a824e80db2a0-kube-api-access-ppnxm\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:18:12.912968 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.912945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8186d25-2a7e-4117-bd1a-5023ed575ec1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-56xpd\" (UID: \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:12.923316 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:12.923287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7z2\" (UniqueName: \"kubernetes.io/projected/e8186d25-2a7e-4117-bd1a-5023ed575ec1-kube-api-access-hd7z2\") pod \"kuadrant-operator-controller-manager-68cf589589-56xpd\" (UID: \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:13.038016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.037927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:13.183070 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.183043 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd"] Jul 31 00:18:13.185255 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:18:13.185230 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8186d25_2a7e_4117_bd1a_5023ed575ec1.slice/crio-69392462d5c70a4b7eda1ee0638d2442bba58ffbe42842fd70e5ec1ee6df0b67 WatchSource:0}: Error finding container 69392462d5c70a4b7eda1ee0638d2442bba58ffbe42842fd70e5ec1ee6df0b67: Status 404 returned error can't find the container with id 69392462d5c70a4b7eda1ee0638d2442bba58ffbe42842fd70e5ec1ee6df0b67 Jul 31 00:18:13.684283 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.684253 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" Jul 31 00:18:13.684283 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.684272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn" event={"ID":"508bfa7d-b73b-4c90-94d6-a824e80db2a0","Type":"ContainerDied","Data":"07346710a7003c6f4bd4c8e75be8ff2e1ec0ed41e82e0e097508603839d5df5c"} Jul 31 00:18:13.684798 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.684337 2575 scope.go:117] "RemoveContainer" containerID="6d80103403205bbf53257663ebdd8ce3e6fa10d213036f8580226ad39888c684" Jul 31 00:18:13.686049 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.686023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" event={"ID":"e8186d25-2a7e-4117-bd1a-5023ed575ec1","Type":"ContainerStarted","Data":"051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047"} Jul 31 00:18:13.686197 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.686073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" event={"ID":"e8186d25-2a7e-4117-bd1a-5023ed575ec1","Type":"ContainerStarted","Data":"69392462d5c70a4b7eda1ee0638d2442bba58ffbe42842fd70e5ec1ee6df0b67"} Jul 31 00:18:13.686197 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.686147 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:13.723814 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.723760 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" podStartSLOduration=1.72373977 podStartE2EDuration="1.72373977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:18:13.721098647 +0000 UTC m=+703.244066391" watchObservedRunningTime="2026-07-31 00:18:13.72373977 +0000 UTC m=+703.246707516" Jul 31 00:18:13.743316 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.743281 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn"] Jul 31 00:18:13.747742 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:13.747710 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-775tn"] Jul 31 00:18:15.048185 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:15.048152 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508bfa7d-b73b-4c90-94d6-a824e80db2a0" path="/var/lib/kubelet/pods/508bfa7d-b73b-4c90-94d6-a824e80db2a0/volumes" Jul 31 00:18:24.695408 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:24.695328 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:18:28.866232 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:28.866200 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:29.204057 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:29.204026 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:32.084839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.084804 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:32.187195 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.187165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:32.287456 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.287423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:32.440935 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.440893 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:18:32.441346 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.441327 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508bfa7d-b73b-4c90-94d6-a824e80db2a0" containerName="manager" Jul 31 00:18:32.441448 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.441349 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="508bfa7d-b73b-4c90-94d6-a824e80db2a0" containerName="manager" Jul 31 00:18:32.441516 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.441501 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="508bfa7d-b73b-4c90-94d6-a824e80db2a0" containerName="manager" Jul 31 00:18:32.443786 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.443764 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:32.446208 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.446188 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 00:18:32.446431 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.446412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbbw\" (UniqueName: \"kubernetes.io/projected/c8d2e371-1816-448e-96e2-76f25edebe39-kube-api-access-fvbbw\") pod \"limitador-limitador-85d9db4d5d-6pkb6\" (UID: \"c8d2e371-1816-448e-96e2-76f25edebe39\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:32.446582 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.446563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c8d2e371-1816-448e-96e2-76f25edebe39-config-file\") pod \"limitador-limitador-85d9db4d5d-6pkb6\" (UID: \"c8d2e371-1816-448e-96e2-76f25edebe39\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:32.451462 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.451430 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:18:32.476253 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.476219 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:18:32.547427 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.547389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c8d2e371-1816-448e-96e2-76f25edebe39-config-file\") pod \"limitador-limitador-85d9db4d5d-6pkb6\" (UID: \"c8d2e371-1816-448e-96e2-76f25edebe39\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:32.547624 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.547452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbbw\" (UniqueName: \"kubernetes.io/projected/c8d2e371-1816-448e-96e2-76f25edebe39-kube-api-access-fvbbw\") pod \"limitador-limitador-85d9db4d5d-6pkb6\" (UID: \"c8d2e371-1816-448e-96e2-76f25edebe39\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:32.548002 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.547984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c8d2e371-1816-448e-96e2-76f25edebe39-config-file\") pod \"limitador-limitador-85d9db4d5d-6pkb6\" (UID: \"c8d2e371-1816-448e-96e2-76f25edebe39\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:32.555502 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.555481 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbbw\" (UniqueName: \"kubernetes.io/projected/c8d2e371-1816-448e-96e2-76f25edebe39-kube-api-access-fvbbw\") pod \"limitador-limitador-85d9db4d5d-6pkb6\" (UID: \"c8d2e371-1816-448e-96e2-76f25edebe39\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:32.754632 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.754553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:32.890542 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.890357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:32.890542 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.890522 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:18:32.893150 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:18:32.893109 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d2e371_1816_448e_96e2_76f25edebe39.slice/crio-a03e51c2af49e0c05091380e0221587ce5047ee58f84e79a9225c5fedb2cbf63 WatchSource:0}: Error finding container a03e51c2af49e0c05091380e0221587ce5047ee58f84e79a9225c5fedb2cbf63: Status 404 returned error can't find the container with id a03e51c2af49e0c05091380e0221587ce5047ee58f84e79a9225c5fedb2cbf63 Jul 31 00:18:32.983561 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.983527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:32.987267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.987238 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-2c4tp"] Jul 31 00:18:32.990512 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.990481 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" Jul 31 00:18:32.992885 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.992860 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vn8lf\"" Jul 31 00:18:32.997618 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:32.997593 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-2c4tp"] Jul 31 00:18:33.051362 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:33.051046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjrk\" (UniqueName: \"kubernetes.io/projected/e091792f-fe89-4e6b-917b-e186a086ab8c-kube-api-access-chjrk\") pod \"authorino-c548b5d7d-2c4tp\" (UID: \"e091792f-fe89-4e6b-917b-e186a086ab8c\") " pod="kuadrant-system/authorino-c548b5d7d-2c4tp" Jul 31 00:18:33.151896 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:33.151841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjrk\" (UniqueName: \"kubernetes.io/projected/e091792f-fe89-4e6b-917b-e186a086ab8c-kube-api-access-chjrk\") pod \"authorino-c548b5d7d-2c4tp\" (UID: \"e091792f-fe89-4e6b-917b-e186a086ab8c\") " pod="kuadrant-system/authorino-c548b5d7d-2c4tp" Jul 31 00:18:33.159663 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:33.159627 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjrk\" (UniqueName: \"kubernetes.io/projected/e091792f-fe89-4e6b-917b-e186a086ab8c-kube-api-access-chjrk\") pod \"authorino-c548b5d7d-2c4tp\" (UID: \"e091792f-fe89-4e6b-917b-e186a086ab8c\") " pod="kuadrant-system/authorino-c548b5d7d-2c4tp" Jul 31 00:18:33.301204 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:33.301166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" Jul 31 00:18:33.432790 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:33.432714 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-2c4tp"] Jul 31 00:18:33.435014 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:18:33.434989 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode091792f_fe89_4e6b_917b_e186a086ab8c.slice/crio-7a640c2fdd504964fa73ea2c0db1aa19bc2b39d4478e99ec8c9a205b893a55c7 WatchSource:0}: Error finding container 7a640c2fdd504964fa73ea2c0db1aa19bc2b39d4478e99ec8c9a205b893a55c7: Status 404 returned error can't find the container with id 7a640c2fdd504964fa73ea2c0db1aa19bc2b39d4478e99ec8c9a205b893a55c7 Jul 31 00:18:33.792418 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:33.792372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" event={"ID":"e091792f-fe89-4e6b-917b-e186a086ab8c","Type":"ContainerStarted","Data":"7a640c2fdd504964fa73ea2c0db1aa19bc2b39d4478e99ec8c9a205b893a55c7"} Jul 31 00:18:33.793956 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:33.793922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" event={"ID":"c8d2e371-1816-448e-96e2-76f25edebe39","Type":"ContainerStarted","Data":"a03e51c2af49e0c05091380e0221587ce5047ee58f84e79a9225c5fedb2cbf63"} Jul 31 00:18:34.802455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:34.802413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" event={"ID":"c8d2e371-1816-448e-96e2-76f25edebe39","Type":"ContainerStarted","Data":"3026d583a17a00c1e7b997ea3fd7b5688ea2c24da3dad8498a947d5933d6dcad"} Jul 31 00:18:34.802886 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:34.802516 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:34.817770 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:34.817711 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" podStartSLOduration=1.627694867 podStartE2EDuration="2.817690561s" totalImagesPullingTime="1.189995694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:18:34.817223405 +0000 UTC m=+724.340191175" watchObservedRunningTime="2026-07-31 00:18:34.817690561 +0000 UTC m=+724.340658308" Jul 31 00:18:35.809494 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:35.809460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" event={"ID":"e091792f-fe89-4e6b-917b-e186a086ab8c","Type":"ContainerStarted","Data":"25e19fb3be09291be58d3b188037871d1176dd0c3b1aa40b8038a070588a1e1b"} Jul 31 00:18:35.823900 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:35.823841 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" podStartSLOduration=1.682213062 podStartE2EDuration="3.823824004s" totalImagesPullingTime="2.141610942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:18:35.822946716 +0000 UTC m=+725.345914463" watchObservedRunningTime="2026-07-31 00:18:35.823824004 +0000 UTC m=+725.346791750" Jul 31 00:18:37.427346 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:37.427315 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:37.527555 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:37.527504 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:45.810438 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:45.810403 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6" Jul 31 00:18:47.758005 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:47.757963 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:47.861250 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:47.861215 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:18:48.073483 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:18:48.073397 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:00.070485 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:00.070449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:00.240808 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:00.240758 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:02.789171 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:02.789108 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-2c4tp"] Jul 31 00:19:02.789640 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:02.789443 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" podUID="e091792f-fe89-4e6b-917b-e186a086ab8c" containerName="authorino" containerID="cri-o://25e19fb3be09291be58d3b188037871d1176dd0c3b1aa40b8038a070588a1e1b" gracePeriod=30 Jul 31 00:19:02.940882 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:02.940851 2575 generic.go:356] "Generic (PLEG): container finished" podID="e091792f-fe89-4e6b-917b-e186a086ab8c" containerID="25e19fb3be09291be58d3b188037871d1176dd0c3b1aa40b8038a070588a1e1b" exitCode=0 Jul 31 00:19:02.941084 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:02.940934 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" event={"ID":"e091792f-fe89-4e6b-917b-e186a086ab8c","Type":"ContainerDied","Data":"25e19fb3be09291be58d3b188037871d1176dd0c3b1aa40b8038a070588a1e1b"} Jul 31 00:19:03.043451 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.043391 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" Jul 31 00:19:03.124280 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.124247 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chjrk\" (UniqueName: \"kubernetes.io/projected/e091792f-fe89-4e6b-917b-e186a086ab8c-kube-api-access-chjrk\") pod \"e091792f-fe89-4e6b-917b-e186a086ab8c\" (UID: \"e091792f-fe89-4e6b-917b-e186a086ab8c\") " Jul 31 00:19:03.126327 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.126298 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e091792f-fe89-4e6b-917b-e186a086ab8c-kube-api-access-chjrk" (OuterVolumeSpecName: "kube-api-access-chjrk") pod "e091792f-fe89-4e6b-917b-e186a086ab8c" (UID: "e091792f-fe89-4e6b-917b-e186a086ab8c"). InnerVolumeSpecName "kube-api-access-chjrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:19:03.225509 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.225474 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chjrk\" (UniqueName: \"kubernetes.io/projected/e091792f-fe89-4e6b-917b-e186a086ab8c-kube-api-access-chjrk\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:19:03.947463 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.947435 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" Jul 31 00:19:03.947852 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.947433 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-2c4tp" event={"ID":"e091792f-fe89-4e6b-917b-e186a086ab8c","Type":"ContainerDied","Data":"7a640c2fdd504964fa73ea2c0db1aa19bc2b39d4478e99ec8c9a205b893a55c7"} Jul 31 00:19:03.947852 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.947556 2575 scope.go:117] "RemoveContainer" containerID="25e19fb3be09291be58d3b188037871d1176dd0c3b1aa40b8038a070588a1e1b" Jul 31 00:19:03.968983 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.968949 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-2c4tp"] Jul 31 00:19:03.972484 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:03.972458 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-2c4tp"] Jul 31 00:19:05.049492 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:05.049455 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e091792f-fe89-4e6b-917b-e186a086ab8c" path="/var/lib/kubelet/pods/e091792f-fe89-4e6b-917b-e186a086ab8c/volumes" Jul 31 00:19:07.564914 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:07.564870 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:07.663986 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:07.663958 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:07.714707 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:07.714672 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:08.445609 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:08.445557 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:11.355839 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:11.355802 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:11.450236 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:11.450194 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:42.908242 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:42.908204 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:43.008996 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:43.008964 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:44.420386 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:44.420355 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:44.673498 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:44.673388 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:44.997542 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:44.997460 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:45.096354 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:45.096311 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:45.165793 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:45.165757 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:19:54.829684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:54.829569 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:19:54.930392 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:19:54.930342 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:20:13.695173 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:13.695137 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:20:14.085258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:14.085174 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:20:23.787436 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:23.787403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:20:24.488556 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:24.488514 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:20:35.016809 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:35.016755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:20:35.891145 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:35.891095 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:20:43.994896 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:43.994854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:20:45.190310 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:45.190269 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:20:58.782036 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.782002 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2"] Jul 31 00:20:58.782442 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.782399 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e091792f-fe89-4e6b-917b-e186a086ab8c" containerName="authorino" Jul 31 00:20:58.782442 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.782410 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e091792f-fe89-4e6b-917b-e186a086ab8c" containerName="authorino" Jul 31 00:20:58.782516 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.782478 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e091792f-fe89-4e6b-917b-e186a086ab8c" containerName="authorino" Jul 31 00:20:58.785586 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.785561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.788192 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.788168 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 31 00:20:58.788192 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.788173 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f2gmq\"" Jul 31 00:20:58.789602 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.789585 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 31 00:20:58.789602 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.789594 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 31 00:20:58.795281 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.795256 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2"] Jul 31 00:20:58.809311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.809269 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:20:58.845783 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.845749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.845973 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.845799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.845973 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.845829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t48r\" (UniqueName: \"kubernetes.io/projected/103658b2-0dba-42fe-b0a5-fdc1aa8735de-kube-api-access-6t48r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.845973 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.845872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.845973 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.845945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.846194 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.846013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103658b2-0dba-42fe-b0a5-fdc1aa8735de-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.946958 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.946915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.946958 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.946960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103658b2-0dba-42fe-b0a5-fdc1aa8735de-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.947260 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.946990 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.947260 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.947009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.947260 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.947026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t48r\" (UniqueName: \"kubernetes.io/projected/103658b2-0dba-42fe-b0a5-fdc1aa8735de-kube-api-access-6t48r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.947260 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.947064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.947518 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.947494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.947574 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.947502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.947574 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.947551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.949317 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.949294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103658b2-0dba-42fe-b0a5-fdc1aa8735de-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.949687 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.949658 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103658b2-0dba-42fe-b0a5-fdc1aa8735de-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:58.954716 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:58.954690 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t48r\" (UniqueName: \"kubernetes.io/projected/103658b2-0dba-42fe-b0a5-fdc1aa8735de-kube-api-access-6t48r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4pkv2\" (UID: \"103658b2-0dba-42fe-b0a5-fdc1aa8735de\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:59.096284 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:59.096192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:20:59.232247 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:59.232216 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2"] Jul 31 00:20:59.234253 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:20:59.234222 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103658b2_0dba_42fe_b0a5_fdc1aa8735de.slice/crio-091361b9dcb5b5e72d9ad9bcda63f3e2e8c344b87823fb4e7d93291fbd96ae94 WatchSource:0}: Error finding container 091361b9dcb5b5e72d9ad9bcda63f3e2e8c344b87823fb4e7d93291fbd96ae94: Status 404 returned error can't find the container with id 091361b9dcb5b5e72d9ad9bcda63f3e2e8c344b87823fb4e7d93291fbd96ae94 Jul 31 00:20:59.235605 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:59.235585 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 00:20:59.519220 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:20:59.519177 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" event={"ID":"103658b2-0dba-42fe-b0a5-fdc1aa8735de","Type":"ContainerStarted","Data":"091361b9dcb5b5e72d9ad9bcda63f3e2e8c344b87823fb4e7d93291fbd96ae94"} Jul 31 00:21:00.526585 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:00.526484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" event={"ID":"103658b2-0dba-42fe-b0a5-fdc1aa8735de","Type":"ContainerStarted","Data":"8335c84535f321df65f4b38f1ee7874f7a34e20acf607b20dd33117124ae232e"} Jul 31 00:21:00.545194 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:00.545113 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" podStartSLOduration=1.510885018 podStartE2EDuration="2.545088013s" totalImagesPullingTime="1.034202995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:21:00.54269939 +0000 UTC m=+870.065667151" watchObservedRunningTime="2026-07-31 00:21:00.545088013 +0000 UTC m=+870.068055759" Jul 31 00:21:01.081895 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:01.081858 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:21:06.787787 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.787750 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5"] Jul 31 00:21:06.792058 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.792031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:06.794484 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.794461 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 31 00:21:06.800834 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.800806 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5"] Jul 31 00:21:06.823418 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.823383 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:21:06.923204 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.923173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:06.923377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.923221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g965\" (UniqueName: \"kubernetes.io/projected/a3aebfc2-17ae-4299-982b-c154479165d0-kube-api-access-6g965\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:06.923377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.923249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:06.923377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.923265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3aebfc2-17ae-4299-982b-c154479165d0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:06.923377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.923308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:06.923377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:06.923327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.024797 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.024747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.024797 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.024801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3aebfc2-17ae-4299-982b-c154479165d0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.025031 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.024867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.025031 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.024889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.025031 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.024932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.025031 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.024966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g965\" (UniqueName: \"kubernetes.io/projected/a3aebfc2-17ae-4299-982b-c154479165d0-kube-api-access-6g965\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.025275 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.025249 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.025334 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.025283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.025380 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.025329 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.027343 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.027304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3aebfc2-17ae-4299-982b-c154479165d0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.027566 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.027549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3aebfc2-17ae-4299-982b-c154479165d0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.033731 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.033704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g965\" (UniqueName: \"kubernetes.io/projected/a3aebfc2-17ae-4299-982b-c154479165d0-kube-api-access-6g965\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5\" (UID: \"a3aebfc2-17ae-4299-982b-c154479165d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.103709 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.103620 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:07.242049 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.242017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5"] Jul 31 00:21:07.244473 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:21:07.244430 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3aebfc2_17ae_4299_982b_c154479165d0.slice/crio-cb9ac9dd31c6793a5490e4be448ccfd4358dc5d287c948342f40d22a33882ed5 WatchSource:0}: Error finding container cb9ac9dd31c6793a5490e4be448ccfd4358dc5d287c948342f40d22a33882ed5: Status 404 returned error can't find the container with id cb9ac9dd31c6793a5490e4be448ccfd4358dc5d287c948342f40d22a33882ed5 Jul 31 00:21:07.565928 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.565894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" event={"ID":"a3aebfc2-17ae-4299-982b-c154479165d0","Type":"ContainerStarted","Data":"11dfafdf3c3c496d659ccda569bdc22f39963ff21be9af0608c155ebec3fa950"} Jul 31 00:21:07.566146 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.565939 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" event={"ID":"a3aebfc2-17ae-4299-982b-c154479165d0","Type":"ContainerStarted","Data":"cb9ac9dd31c6793a5490e4be448ccfd4358dc5d287c948342f40d22a33882ed5"} Jul 31 00:21:07.582267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:07.582214 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" podStartSLOduration=1.3925595610000001 podStartE2EDuration="1.582194935s" totalImagesPullingTime="189.635374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:21:07.580857792 +0000 UTC m=+877.103825528" watchObservedRunningTime="2026-07-31 00:21:07.582194935 +0000 UTC m=+877.105162680" Jul 31 00:21:08.983545 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:08.983505 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:21:09.096722 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:09.096691 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:21:09.096902 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:09.096770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:21:09.109343 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:09.109314 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:21:09.594203 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:09.594173 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4pkv2" Jul 31 00:21:10.249477 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:10.249442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:21:17.104391 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:17.104350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:17.104958 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:17.104443 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:17.117262 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:17.117230 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:17.633143 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:17.633089 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5" Jul 31 00:21:42.060466 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:42.060427 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:21:50.124660 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:50.124625 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:21:50.224098 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:50.224066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:21:51.281520 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:51.281455 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:21:51.374963 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:51.374928 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:21:54.985581 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:54.985548 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:21:55.033443 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:21:55.033406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:22:18.987914 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:22:18.987873 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:22:19.538005 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:22:19.537947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:22:19.638094 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:22:19.638057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:22:28.976538 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:22:28.976492 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:22:39.670381 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:22:39.670333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:22:39.720152 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:22:39.720090 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:22:40.086791 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:22:40.086711 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:23:32.313543 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:23:32.313505 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:23:56.665159 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:23:56.665096 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:23:58.195377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:23:58.195329 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:24:21.883132 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:24:21.883086 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:24:23.164772 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:24:23.164696 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:24:31.000709 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:24:31.000675 2575 scope.go:117] "RemoveContainer" containerID="9f560f3f7b63cb8c4f76c9a168842bf2d499116900aa866e7a0e887e169f15bb" Jul 31 00:24:31.008934 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:24:31.008920 2575 scope.go:117] "RemoveContainer" containerID="5573fceb686f35eb930aefa9507dd76d053e57a1bc6e9604ea799705b8a4466b" Jul 31 00:24:32.777294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:24:32.777261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:25:08.387619 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:08.387578 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:25:08.488372 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:08.488341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:25:08.588291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:08.588257 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:25:14.582411 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:14.582378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:25:14.682304 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:14.682269 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:25:14.819964 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:14.819933 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:25:20.319811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:20.319772 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:25:20.435913 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:20.435875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:25:36.173439 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:36.173400 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:25:43.276090 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:43.276050 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:25:58.677569 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:25:58.677485 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:26:05.981006 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:26:05.980966 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:26:15.081393 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:26:15.081354 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:26:22.385134 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:26:22.385086 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:26:34.785363 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:26:34.785325 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:26:39.876653 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:26:39.876606 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:26:48.380510 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:26:48.380470 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:26:57.086362 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:26:57.086322 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:27:10.182236 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:27:10.182195 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:27:21.076342 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:27:21.076304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:27:43.386462 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:27:43.386375 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:27:53.781025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:27:53.780989 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:27:59.679203 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:27:59.679160 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:28:07.579276 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:28:07.579235 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:28:15.386248 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:28:15.386210 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:28:20.688333 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:28:20.688297 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:28:25.686705 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:28:25.686668 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:28:33.084797 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:28:33.084759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:28:44.587812 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:28:44.587774 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:28:52.184876 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:28:52.184831 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:29:02.782789 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:29:02.782751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:29:07.890631 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:29:07.890592 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:29:14.278798 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:29:14.278759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:29:23.186393 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:29:23.186361 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:29:28.483055 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:29:28.483018 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:29:36.607499 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:29:36.607449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:29:40.103287 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:29:40.103255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:30:34.453728 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.453636 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd"] Jul 31 00:30:34.454301 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.453847 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" podUID="e8186d25-2a7e-4117-bd1a-5023ed575ec1" containerName="manager" containerID="cri-o://051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047" gracePeriod=10 Jul 31 00:30:34.711828 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.711751 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:30:34.747040 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.747004 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8186d25-2a7e-4117-bd1a-5023ed575ec1-extensions-socket-volume\") pod \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\" (UID: \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\") " Jul 31 00:30:34.747236 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.747059 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd7z2\" (UniqueName: \"kubernetes.io/projected/e8186d25-2a7e-4117-bd1a-5023ed575ec1-kube-api-access-hd7z2\") pod \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\" (UID: \"e8186d25-2a7e-4117-bd1a-5023ed575ec1\") " Jul 31 00:30:34.747449 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.747424 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8186d25-2a7e-4117-bd1a-5023ed575ec1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e8186d25-2a7e-4117-bd1a-5023ed575ec1" (UID: "e8186d25-2a7e-4117-bd1a-5023ed575ec1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:30:34.749267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.749239 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8186d25-2a7e-4117-bd1a-5023ed575ec1-kube-api-access-hd7z2" (OuterVolumeSpecName: "kube-api-access-hd7z2") pod "e8186d25-2a7e-4117-bd1a-5023ed575ec1" (UID: "e8186d25-2a7e-4117-bd1a-5023ed575ec1"). InnerVolumeSpecName "kube-api-access-hd7z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:30:34.847692 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.847647 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8186d25-2a7e-4117-bd1a-5023ed575ec1-extensions-socket-volume\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:30:34.847692 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:34.847687 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd7z2\" (UniqueName: \"kubernetes.io/projected/e8186d25-2a7e-4117-bd1a-5023ed575ec1-kube-api-access-hd7z2\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:30:35.353954 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.353860 2575 generic.go:356] "Generic (PLEG): container finished" podID="e8186d25-2a7e-4117-bd1a-5023ed575ec1" containerID="051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047" exitCode=0 Jul 31 00:30:35.353954 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.353917 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" Jul 31 00:30:35.354164 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.353917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" event={"ID":"e8186d25-2a7e-4117-bd1a-5023ed575ec1","Type":"ContainerDied","Data":"051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047"} Jul 31 00:30:35.354164 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.354037 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" event={"ID":"e8186d25-2a7e-4117-bd1a-5023ed575ec1","Type":"ContainerDied","Data":"69392462d5c70a4b7eda1ee0638d2442bba58ffbe42842fd70e5ec1ee6df0b67"} Jul 31 00:30:35.354164 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.354063 2575 scope.go:117] "RemoveContainer" containerID="051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047" Jul 31 00:30:35.364063 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.364044 2575 scope.go:117] "RemoveContainer" containerID="051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047" Jul 31 00:30:35.364387 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:30:35.364365 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047\": container with ID starting with 051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047 not found: ID does not exist" containerID="051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047" Jul 31 00:30:35.364452 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.364400 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047"} err="failed to get container status \"051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047\": rpc error: code = NotFound desc = could not find container \"051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047\": container with ID starting with 051bbf2c01941f51907deb43dcc8c88a77a4c43fdcc7e0856d785c5e08d51047 not found: ID does not exist" Jul 31 00:30:35.372942 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.372913 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd"] Jul 31 00:30:35.376355 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.376331 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd"] Jul 31 00:30:35.694215 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:35.694157 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-56xpd" podUID="e8186d25-2a7e-4117-bd1a-5023ed575ec1" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 31 00:30:37.048686 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:30:37.048651 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8186d25-2a7e-4117-bd1a-5023ed575ec1" path="/var/lib/kubelet/pods/e8186d25-2a7e-4117-bd1a-5023ed575ec1/volumes" Jul 31 00:31:25.096653 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:25.096608 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:31:35.187334 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.187299 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g"] Jul 31 00:31:35.187740 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.187640 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8186d25-2a7e-4117-bd1a-5023ed575ec1" containerName="manager" Jul 31 00:31:35.187740 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.187650 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8186d25-2a7e-4117-bd1a-5023ed575ec1" containerName="manager" Jul 31 00:31:35.187740 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.187706 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8186d25-2a7e-4117-bd1a-5023ed575ec1" containerName="manager" Jul 31 00:31:35.190847 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.190829 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:35.193501 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.193481 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r29qw\"" Jul 31 00:31:35.201057 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.201032 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g"] Jul 31 00:31:35.282693 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.282655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzqx\" (UniqueName: \"kubernetes.io/projected/2f1e0077-19ab-4795-a123-2e0f34e4e93a-kube-api-access-nkzqx\") pod \"kuadrant-operator-controller-manager-68cf589589-qrj2g\" (UID: \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:35.282867 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.282714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f1e0077-19ab-4795-a123-2e0f34e4e93a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qrj2g\" (UID: \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:35.384233 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.384194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f1e0077-19ab-4795-a123-2e0f34e4e93a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qrj2g\" (UID: \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:35.384406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.384299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzqx\" (UniqueName: \"kubernetes.io/projected/2f1e0077-19ab-4795-a123-2e0f34e4e93a-kube-api-access-nkzqx\") pod \"kuadrant-operator-controller-manager-68cf589589-qrj2g\" (UID: \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:35.384609 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.384579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f1e0077-19ab-4795-a123-2e0f34e4e93a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-qrj2g\" (UID: \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:35.393589 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.393559 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzqx\" (UniqueName: \"kubernetes.io/projected/2f1e0077-19ab-4795-a123-2e0f34e4e93a-kube-api-access-nkzqx\") pod \"kuadrant-operator-controller-manager-68cf589589-qrj2g\" (UID: \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:35.502335 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.502255 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:35.646644 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:35.646622 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g"] Jul 31 00:31:35.648956 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:31:35.648917 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1e0077_19ab_4795_a123_2e0f34e4e93a.slice/crio-0e6c163a0afe7faf4301b5c24aff97aa285e758f0855696c9642fd9edfce421e WatchSource:0}: Error finding container 0e6c163a0afe7faf4301b5c24aff97aa285e758f0855696c9642fd9edfce421e: Status 404 returned error can't find the container with id 0e6c163a0afe7faf4301b5c24aff97aa285e758f0855696c9642fd9edfce421e Jul 31 00:31:36.651323 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:36.651288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" event={"ID":"2f1e0077-19ab-4795-a123-2e0f34e4e93a","Type":"ContainerStarted","Data":"fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5"} Jul 31 00:31:36.651705 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:36.651339 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" event={"ID":"2f1e0077-19ab-4795-a123-2e0f34e4e93a","Type":"ContainerStarted","Data":"0e6c163a0afe7faf4301b5c24aff97aa285e758f0855696c9642fd9edfce421e"} Jul 31 00:31:36.651705 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:36.651373 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:36.676368 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:36.676310 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" podStartSLOduration=1.676293099 podStartE2EDuration="1.676293099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:31:36.674723013 +0000 UTC m=+1506.197690760" watchObservedRunningTime="2026-07-31 00:31:36.676293099 +0000 UTC m=+1506.199260843" Jul 31 00:31:47.657741 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:47.657703 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:31:54.275719 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:54.275634 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:31:59.377640 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:31:59.377601 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:32:23.985797 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:32:23.985763 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:32:34.582801 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:32:34.582760 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:32:42.479158 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:32:42.479104 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:32:46.011448 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:32:46.011412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:32:46.112490 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:32:46.112429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:32:50.180178 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:32:50.180148 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:32:55.090624 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:32:55.090590 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:33:05.678748 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:33:05.678707 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:33:12.017666 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:33:12.017631 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:33:20.885946 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:33:20.885907 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:33:25.887333 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:33:25.887250 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:33:36.986529 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:33:36.986491 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:33:48.285012 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:33:48.284970 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:33:55.278948 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:33:55.278913 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:34:16.485392 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:16.485359 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:34:21.681179 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:21.681135 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:34:30.004037 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:30.004001 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:34:33.074187 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:33.074156 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:34:52.789524 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.789436 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl"] Jul 31 00:34:52.793089 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.793067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:52.795593 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.795574 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-209b6258-kserve-self-signed-certs\"" Jul 31 00:34:52.804026 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.804002 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl"] Jul 31 00:34:52.816248 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.816219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:34:52.942090 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.942049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfh25\" (UniqueName: \"kubernetes.io/projected/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-kube-api-access-qfh25\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:52.942336 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.942150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-model-cache\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:52.942336 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.942178 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tmp-dir\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:52.942336 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.942246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-dshm\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:52.942336 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.942301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-home\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:52.942496 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:52.942370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tls-certs\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.043829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.043732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfh25\" (UniqueName: \"kubernetes.io/projected/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-kube-api-access-qfh25\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.043829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.043808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-model-cache\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.044077 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.043833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tmp-dir\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.044077 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.043869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-dshm\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.044077 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.044026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-home\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.044267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.044084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tls-certs\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.044267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.044230 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-model-cache\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.044267 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.044251 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tmp-dir\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.044474 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.044456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-home\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.046352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.046326 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-dshm\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.046968 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.046948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tls-certs\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.051933 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.051907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfh25\" (UniqueName: \"kubernetes.io/projected/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-kube-api-access-qfh25\") pod \"e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.102259 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.102220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:34:53.240857 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.240794 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl"] Jul 31 00:34:53.244142 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:34:53.244096 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeec8987e_3cd1_4aa4_bcaa_87026b3d8c53.slice/crio-8d21584bc66c213fc9bc832f642233b86b31606912073d0259bb7d0ed4caa3f5 WatchSource:0}: Error finding container 8d21584bc66c213fc9bc832f642233b86b31606912073d0259bb7d0ed4caa3f5: Status 404 returned error can't find the container with id 8d21584bc66c213fc9bc832f642233b86b31606912073d0259bb7d0ed4caa3f5 Jul 31 00:34:53.600556 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.600448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" event={"ID":"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53","Type":"ContainerStarted","Data":"1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c"} Jul 31 00:34:53.600556 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.600497 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" event={"ID":"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53","Type":"ContainerStarted","Data":"8d21584bc66c213fc9bc832f642233b86b31606912073d0259bb7d0ed4caa3f5"} Jul 31 00:34:53.617339 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:53.617286 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" podStartSLOduration=1.6172695419999998 podStartE2EDuration="1.617269542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:34:53.615927626 +0000 UTC m=+1703.138895395" watchObservedRunningTime="2026-07-31 00:34:53.617269542 +0000 UTC m=+1703.140237286" Jul 31 00:34:55.384338 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:34:55.384296 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:35:03.102503 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:03.102466 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:35:03.102960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:03.102521 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:35:03.115087 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:03.115062 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:35:03.661004 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:03.660972 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:35:35.923574 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:35.923540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:35:37.175692 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:37.175656 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl"] Jul 31 00:35:37.192479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:37.192450 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:35:39.078676 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:39.078636 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:35:48.682389 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:48.682348 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:35:52.195575 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.195513 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" podUID="eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" containerName="main" containerID="cri-o://1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c" gracePeriod=45 Jul 31 00:35:52.455707 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.455640 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:35:52.581623 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.581584 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tls-certs\") pod \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " Jul 31 00:35:52.581623 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.581633 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-home\") pod \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " Jul 31 00:35:52.581887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.581759 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tmp-dir\") pod \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " Jul 31 00:35:52.581887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.581798 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-dshm\") pod \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " Jul 31 00:35:52.581887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.581829 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-model-cache\") pod \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " Jul 31 00:35:52.581887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.581856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfh25\" (UniqueName: \"kubernetes.io/projected/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-kube-api-access-qfh25\") pod \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\" (UID: \"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53\") " Jul 31 00:35:52.582095 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.581950 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-home" (OuterVolumeSpecName: "home") pod "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" (UID: "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:35:52.582095 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.581980 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" (UID: "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:35:52.582232 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.582151 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-model-cache" (OuterVolumeSpecName: "model-cache") pod "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" (UID: "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:35:52.582232 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.582194 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-home\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:35:52.582232 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.582208 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tmp-dir\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:35:52.583988 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.583961 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-dshm" (OuterVolumeSpecName: "dshm") pod "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" (UID: "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:35:52.584089 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.584008 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" (UID: "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:35:52.584089 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.584077 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-kube-api-access-qfh25" (OuterVolumeSpecName: "kube-api-access-qfh25") pod "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" (UID: "eec8987e-3cd1-4aa4-bcaa-87026b3d8c53"). InnerVolumeSpecName "kube-api-access-qfh25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:35:52.682993 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.682956 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-tls-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:35:52.682993 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.682985 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-dshm\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:35:52.682993 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.682994 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-model-cache\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:35:52.682993 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.683002 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfh25\" (UniqueName: \"kubernetes.io/projected/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53-kube-api-access-qfh25\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:35:52.891520 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.891421 2575 generic.go:356] "Generic (PLEG): container finished" podID="eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" containerID="1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c" exitCode=0 Jul 31 00:35:52.891520 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.891464 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" Jul 31 00:35:52.891722 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.891509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" event={"ID":"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53","Type":"ContainerDied","Data":"1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c"} Jul 31 00:35:52.891722 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.891556 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl" event={"ID":"eec8987e-3cd1-4aa4-bcaa-87026b3d8c53","Type":"ContainerDied","Data":"8d21584bc66c213fc9bc832f642233b86b31606912073d0259bb7d0ed4caa3f5"} Jul 31 00:35:52.891722 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.891576 2575 scope.go:117] "RemoveContainer" containerID="1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c" Jul 31 00:35:52.900710 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.900684 2575 scope.go:117] "RemoveContainer" containerID="1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c" Jul 31 00:35:52.900999 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:35:52.900981 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c\": container with ID starting with 1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c not found: ID does not exist" containerID="1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c" Jul 31 00:35:52.901050 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.901008 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c"} err="failed to get container status \"1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c\": rpc error: code = NotFound desc = could not find container \"1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c\": container with ID starting with 1efb5adc272719ed9c737e1f1d4bcc62992065308f10a4e445f0d81e3507bc5c not found: ID does not exist" Jul 31 00:35:52.911744 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.911703 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl"] Jul 31 00:35:52.915584 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:52.915556 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-209b6258-kserve-cb75979dc-njpkl"] Jul 31 00:35:53.050551 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:53.050508 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" path="/var/lib/kubelet/pods/eec8987e-3cd1-4aa4-bcaa-87026b3d8c53/volumes" Jul 31 00:35:55.374602 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:35:55.374563 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:36:01.684103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:36:01.684064 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:36:11.875925 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:36:11.875880 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:36:19.176020 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:36:19.175984 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:36:25.481719 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:36:25.481632 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:36:34.080769 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:36:34.080718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:36:44.475541 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:36:44.475503 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:36:53.079774 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:36:53.079737 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:37:01.382873 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:37:01.382829 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:37:06.984024 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:37:06.983982 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:37:14.480638 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:37:14.480599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:37:25.492112 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:37:25.492068 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:37:33.484759 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:37:33.484720 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:37:40.078011 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:37:40.077973 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:37:44.378082 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:37:44.378041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:37:54.381636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:37:54.381547 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:38:03.276358 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:38:03.276323 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:38:13.180343 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:38:13.180310 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:38:30.490346 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:38:30.490306 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:38:37.582515 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:38:37.582479 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:38:47.676111 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:38:47.676071 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:38:57.045175 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:38:57.045135 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:38:57.135000 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:38:57.134968 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:03.175094 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:03.175055 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:39:10.181974 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:10.181940 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:39:12.411394 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:12.411361 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:12.517587 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:12.517552 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:19.281323 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:19.281275 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:39:27.189671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:27.189587 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:39:35.482805 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:35.482760 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:39:38.456462 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:38.456425 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:40.983797 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:40.983760 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:39:41.582431 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:41.582384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:41.679450 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:41.679407 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:42.024993 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:42.024959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:47.596195 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:47.596159 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:47.645648 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:47.645615 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:47.826730 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:47.826693 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:48.130099 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:48.130056 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:48.414772 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:48.414735 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:50.645796 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:50.645763 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:50.745059 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:50.745025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:51.755091 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:51.755052 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:52.226723 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:52.226687 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:52.322935 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:52.322903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:52.506019 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:52.505932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:39:54.585061 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:39:54.585025 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:40:16.227729 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.227682 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6"] Jul 31 00:40:16.228359 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.228227 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" containerName="main" Jul 31 00:40:16.228359 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.228249 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" containerName="main" Jul 31 00:40:16.228492 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.228364 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec8987e-3cd1-4aa4-bcaa-87026b3d8c53" containerName="main" Jul 31 00:40:16.231423 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.231398 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" Jul 31 00:40:16.233897 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.233872 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gpvxj\"" Jul 31 00:40:16.234051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.233939 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 00:40:16.237916 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.237888 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6"] Jul 31 00:40:16.330388 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.330350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6848d9b9-5a92-4181-8aa3-21302e1824e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6\" (UID: \"6848d9b9-5a92-4181-8aa3-21302e1824e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" Jul 31 00:40:16.431707 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.431665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6848d9b9-5a92-4181-8aa3-21302e1824e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6\" (UID: \"6848d9b9-5a92-4181-8aa3-21302e1824e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" Jul 31 00:40:16.432406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.432378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6848d9b9-5a92-4181-8aa3-21302e1824e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6\" (UID: \"6848d9b9-5a92-4181-8aa3-21302e1824e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" Jul 31 00:40:16.549561 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.549466 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" Jul 31 00:40:16.690446 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.690381 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6"] Jul 31 00:40:16.692840 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:40:16.692812 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6848d9b9_5a92_4181_8aa3_21302e1824e5.slice/crio-424551df51bb14406eec635f931dadc8fc44e678493d7968181c6aab1f917066 WatchSource:0}: Error finding container 424551df51bb14406eec635f931dadc8fc44e678493d7968181c6aab1f917066: Status 404 returned error can't find the container with id 424551df51bb14406eec635f931dadc8fc44e678493d7968181c6aab1f917066 Jul 31 00:40:16.694637 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:16.694617 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 00:40:17.180840 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:17.180800 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" event={"ID":"6848d9b9-5a92-4181-8aa3-21302e1824e5","Type":"ContainerStarted","Data":"424551df51bb14406eec635f931dadc8fc44e678493d7968181c6aab1f917066"} Jul 31 00:40:20.197311 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:20.197278 2575 generic.go:356] "Generic (PLEG): container finished" podID="6848d9b9-5a92-4181-8aa3-21302e1824e5" containerID="ebd6a3cb42b6996fa743919d8a596eddc7104d4123b3387e2de8e621a17460ff" exitCode=0 Jul 31 00:40:20.197712 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:20.197342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" event={"ID":"6848d9b9-5a92-4181-8aa3-21302e1824e5","Type":"ContainerDied","Data":"ebd6a3cb42b6996fa743919d8a596eddc7104d4123b3387e2de8e621a17460ff"} Jul 31 00:40:21.340098 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:21.340073 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" Jul 31 00:40:21.376011 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:21.375978 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6848d9b9-5a92-4181-8aa3-21302e1824e5-maas-api-service-ca\") pod \"6848d9b9-5a92-4181-8aa3-21302e1824e5\" (UID: \"6848d9b9-5a92-4181-8aa3-21302e1824e5\") " Jul 31 00:40:21.376369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:21.376345 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6848d9b9-5a92-4181-8aa3-21302e1824e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6848d9b9-5a92-4181-8aa3-21302e1824e5" (UID: "6848d9b9-5a92-4181-8aa3-21302e1824e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:40:21.477335 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:21.477241 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6848d9b9-5a92-4181-8aa3-21302e1824e5-maas-api-service-ca\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:40:22.208675 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:22.208645 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" Jul 31 00:40:22.208858 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:22.208640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6" event={"ID":"6848d9b9-5a92-4181-8aa3-21302e1824e5","Type":"ContainerDied","Data":"424551df51bb14406eec635f931dadc8fc44e678493d7968181c6aab1f917066"} Jul 31 00:40:22.208858 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:22.208799 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="424551df51bb14406eec635f931dadc8fc44e678493d7968181c6aab1f917066" Jul 31 00:40:32.282979 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:32.282938 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:40:32.447459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:32.447410 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6"] Jul 31 00:40:32.452520 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:32.452485 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2c8c8809-n86p6"] Jul 31 00:40:33.049071 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.049034 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6848d9b9-5a92-4181-8aa3-21302e1824e5" path="/var/lib/kubelet/pods/6848d9b9-5a92-4181-8aa3-21302e1824e5/volumes" Jul 31 00:40:33.563006 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.562965 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89"] Jul 31 00:40:33.563589 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.563567 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6848d9b9-5a92-4181-8aa3-21302e1824e5" containerName="revoke-keys" Jul 31 00:40:33.563680 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.563592 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6848d9b9-5a92-4181-8aa3-21302e1824e5" containerName="revoke-keys" Jul 31 00:40:33.563743 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.563695 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6848d9b9-5a92-4181-8aa3-21302e1824e5" containerName="revoke-keys" Jul 31 00:40:33.570193 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.570163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.573477 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.573441 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5c18c4ee-gw-openshift-default-dockercfg-n4872\"" Jul 31 00:40:33.580577 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.580537 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89"] Jul 31 00:40:33.689605 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.689564 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-certs\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.689605 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.689608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-data\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.689877 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.689681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c2015-2340-4d6d-9894-d73489452454-istiod-ca-cert\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.689877 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.689729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-credential-socket\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.689877 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.689751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-socket\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.689877 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.689785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-envoy\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.689877 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.689873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwh5\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-kube-api-access-wqwh5\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.690058 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.689907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-istio-token\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.690058 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.690002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c2015-2340-4d6d-9894-d73489452454-istio-podinfo\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.790981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-envoy\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwh5\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-kube-api-access-wqwh5\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791342 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-istio-token\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791342 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c2015-2340-4d6d-9894-d73489452454-istio-podinfo\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791342 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-certs\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791342 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-data\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791342 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c2015-2340-4d6d-9894-d73489452454-istiod-ca-cert\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791342 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791247 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-credential-socket\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791342 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-socket\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791692 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-data\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791692 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791658 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-socket\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791807 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791766 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-certs\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.791916 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.791885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-credential-socket\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.792050 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.792000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c2015-2340-4d6d-9894-d73489452454-istiod-ca-cert\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.793451 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.793419 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-envoy\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.793712 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.793689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c2015-2340-4d6d-9894-d73489452454-istio-podinfo\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.800662 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.800621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-istio-token\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.800820 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.800737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwh5\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-kube-api-access-wqwh5\") pod \"e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:33.807369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.807340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:40:33.890907 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:33.890808 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:34.050105 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.050074 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89"] Jul 31 00:40:34.052343 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:40:34.052308 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719c2015_2340_4d6d_9894_d73489452454.slice/crio-307fdee9c9943281d510bd696b133ca0b1f4f90965ffda519fd47eee76fd1c11 WatchSource:0}: Error finding container 307fdee9c9943281d510bd696b133ca0b1f4f90965ffda519fd47eee76fd1c11: Status 404 returned error can't find the container with id 307fdee9c9943281d510bd696b133ca0b1f4f90965ffda519fd47eee76fd1c11 Jul 31 00:40:34.054620 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.054585 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:40:34.054699 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.054656 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:40:34.054699 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.054686 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:40:34.197941 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.197886 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:40:34.268252 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.268214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" event={"ID":"719c2015-2340-4d6d-9894-d73489452454","Type":"ContainerStarted","Data":"2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d"} Jul 31 00:40:34.268443 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.268278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" event={"ID":"719c2015-2340-4d6d-9894-d73489452454","Type":"ContainerStarted","Data":"307fdee9c9943281d510bd696b133ca0b1f4f90965ffda519fd47eee76fd1c11"} Jul 31 00:40:34.291722 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.291648 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" podStartSLOduration=1.291622411 podStartE2EDuration="1.291622411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:40:34.289225522 +0000 UTC m=+2043.812193289" watchObservedRunningTime="2026-07-31 00:40:34.291622411 +0000 UTC m=+2043.814590159" Jul 31 00:40:34.891880 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:34.891840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:35.674314 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:35.674283 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:40:35.833721 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:35.833686 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:40:35.896857 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:35.896828 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:36.004662 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:36.004571 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:40:36.169524 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:36.169483 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:40:36.282107 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:36.282015 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:36.283097 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:36.283079 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:36.610037 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:36.609953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:40:36.711806 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:36.711774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:40:42.982461 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:42.982416 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:40:49.346886 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:49.346834 2575 generic.go:356] "Generic (PLEG): container finished" podID="719c2015-2340-4d6d-9894-d73489452454" containerID="2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d" exitCode=0 Jul 31 00:40:49.347397 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:49.346917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" event={"ID":"719c2015-2340-4d6d-9894-d73489452454","Type":"ContainerDied","Data":"2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d"} Jul 31 00:40:49.347397 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:49.347366 2575 scope.go:117] "RemoveContainer" containerID="2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d" Jul 31 00:40:49.348450 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:49.348419 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:40:49.348511 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:49.348494 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:40:49.348549 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:49.348527 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:40:50.358515 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:50.358460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" event={"ID":"719c2015-2340-4d6d-9894-d73489452454","Type":"ContainerStarted","Data":"fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2"} Jul 31 00:40:50.891188 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:50.891149 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:50.892491 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:50.892460 2575 patch_prober.go:28] interesting pod/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 31 00:40:50.892627 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:50.892516 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 31 00:40:51.282857 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:51.282817 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:51.891641 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:51.891606 2575 patch_prober.go:28] interesting pod/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 31 00:40:51.892042 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:51.891665 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 31 00:40:52.891589 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:52.891507 2575 patch_prober.go:28] interesting pod/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 31 00:40:52.891589 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:52.891573 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 31 00:40:53.891985 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:53.891945 2575 patch_prober.go:28] interesting pod/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 31 00:40:53.892415 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:53.892018 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 31 00:40:54.892228 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:54.892188 2575 patch_prober.go:28] interesting pod/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 31 00:40:54.892641 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:54.892247 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 31 00:40:55.891551 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:55.891515 2575 patch_prober.go:28] interesting pod/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 31 00:40:55.891748 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:55.891574 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 31 00:40:56.896078 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:56.896041 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:40:56.897093 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:40:56.897069 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:41:16.551549 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:16.551514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:16.660006 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:16.659975 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:16.714439 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:16.714397 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:16.850414 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:16.850320 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89"] Jul 31 00:41:16.850611 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:16.850561 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" containerID="cri-o://fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2" gracePeriod=30 Jul 31 00:41:16.876517 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:16.876484 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:17.110847 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:17.110752 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:17.348970 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:17.348939 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:17.665590 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:17.665550 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:17.749264 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:17.749229 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:19.679721 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:19.679689 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:41:20.170735 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:20.170696 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:20.271330 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:20.271290 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:21.384814 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:21.384783 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:21.841526 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:21.841491 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:21.943293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:21.943254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:22.123474 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.123446 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:41:22.242752 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.242713 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:22.253795 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.253767 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-socket\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.253940 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.253804 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c2015-2340-4d6d-9894-d73489452454-istio-podinfo\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.253940 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.253827 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c2015-2340-4d6d-9894-d73489452454-istiod-ca-cert\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.253940 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.253879 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-istio-token\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.253940 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.253897 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-certs\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.253940 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.253913 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-credential-socket\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.253940 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.253941 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-envoy\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.254293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.253980 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqwh5\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-kube-api-access-wqwh5\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.254293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254027 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-data\") pod \"719c2015-2340-4d6d-9894-d73489452454\" (UID: \"719c2015-2340-4d6d-9894-d73489452454\") " Jul 31 00:41:22.254293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254051 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:41:22.254293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254251 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:41:22.254491 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254304 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719c2015-2340-4d6d-9894-d73489452454-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:41:22.254491 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254401 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.254491 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254422 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c2015-2340-4d6d-9894-d73489452454-istiod-ca-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.254491 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254436 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-workload-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.254491 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254435 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-data" (OuterVolumeSpecName: "istio-data") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:41:22.254667 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.254598 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:41:22.256296 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.256258 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-istio-token" (OuterVolumeSpecName: "istio-token") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:41:22.256457 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.256436 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:41:22.256580 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.256563 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-kube-api-access-wqwh5" (OuterVolumeSpecName: "kube-api-access-wqwh5") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "kube-api-access-wqwh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:41:22.256644 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.256568 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/719c2015-2340-4d6d-9894-d73489452454-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "719c2015-2340-4d6d-9894-d73489452454" (UID: "719c2015-2340-4d6d-9894-d73489452454"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 00:41:22.355363 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.355280 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c2015-2340-4d6d-9894-d73489452454-istio-podinfo\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.355363 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.355309 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-istio-token\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.355363 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.355320 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-credential-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.355363 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.355329 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-envoy\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.355363 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.355338 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwh5\" (UniqueName: \"kubernetes.io/projected/719c2015-2340-4d6d-9894-d73489452454-kube-api-access-wqwh5\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.355363 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.355347 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c2015-2340-4d6d-9894-d73489452454-istio-data\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:22.522875 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.522842 2575 generic.go:356] "Generic (PLEG): container finished" podID="719c2015-2340-4d6d-9894-d73489452454" containerID="fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2" exitCode=0 Jul 31 00:41:22.523272 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.522896 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" Jul 31 00:41:22.523272 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.522924 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" event={"ID":"719c2015-2340-4d6d-9894-d73489452454","Type":"ContainerDied","Data":"fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2"} Jul 31 00:41:22.523272 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.522984 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89" event={"ID":"719c2015-2340-4d6d-9894-d73489452454","Type":"ContainerDied","Data":"307fdee9c9943281d510bd696b133ca0b1f4f90965ffda519fd47eee76fd1c11"} Jul 31 00:41:22.523272 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.523015 2575 scope.go:117] "RemoveContainer" containerID="fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2" Jul 31 00:41:22.533375 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.533353 2575 scope.go:117] "RemoveContainer" containerID="2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d" Jul 31 00:41:22.541635 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.541609 2575 scope.go:117] "RemoveContainer" containerID="fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2" Jul 31 00:41:22.541955 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:41:22.541931 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2\": container with ID starting with fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2 not found: ID does not exist" containerID="fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2" Jul 31 00:41:22.542024 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.541966 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2"} err="failed to get container status \"fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2\": rpc error: code = NotFound desc = could not find container \"fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2\": container with ID starting with fd4b5eede74d6de5465154820b4a4c0b953eeb99fa718ed465171febdb8340f2 not found: ID does not exist" Jul 31 00:41:22.542024 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.541986 2575 scope.go:117] "RemoveContainer" containerID="2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d" Jul 31 00:41:22.542284 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:41:22.542261 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d\": container with ID starting with 2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d not found: ID does not exist" containerID="2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d" Jul 31 00:41:22.542401 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.542287 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d"} err="failed to get container status \"2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d\": rpc error: code = NotFound desc = could not find container \"2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d\": container with ID starting with 2a5ad351627e34b88dd78b0dabb4cf1cb2ca762cc39aa2ffdf6c48e19404169d not found: ID does not exist" Jul 31 00:41:22.544713 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.544690 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89"] Jul 31 00:41:22.549172 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:22.549149 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5c18c4ee-gw-openshift-default-86cd6d7dd8-49v89"] Jul 31 00:41:23.048752 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:23.048713 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="719c2015-2340-4d6d-9894-d73489452454" path="/var/lib/kubelet/pods/719c2015-2340-4d6d-9894-d73489452454/volumes" Jul 31 00:41:28.377226 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:28.377190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:41:46.075976 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.075943 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp"] Jul 31 00:41:46.076408 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.076318 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" Jul 31 00:41:46.076408 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.076329 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" Jul 31 00:41:46.076408 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.076396 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" Jul 31 00:41:46.076408 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.076405 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" Jul 31 00:41:46.079286 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.079264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" Jul 31 00:41:46.081873 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.081852 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gpvxj\"" Jul 31 00:41:46.082007 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.081859 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 00:41:46.085424 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.085401 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp"] Jul 31 00:41:46.167016 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.166969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f81631-6f58-4b80-8923-b2568edaacf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eee00287-m6ltp\" (UID: \"92f81631-6f58-4b80-8923-b2568edaacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" Jul 31 00:41:46.268518 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.268473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f81631-6f58-4b80-8923-b2568edaacf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eee00287-m6ltp\" (UID: \"92f81631-6f58-4b80-8923-b2568edaacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" Jul 31 00:41:46.269153 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.269103 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f81631-6f58-4b80-8923-b2568edaacf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eee00287-m6ltp\" (UID: \"92f81631-6f58-4b80-8923-b2568edaacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" Jul 31 00:41:46.390709 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.390625 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" Jul 31 00:41:46.526296 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.526265 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp"] Jul 31 00:41:46.528186 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:41:46.528147 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f81631_6f58_4b80_8923_b2568edaacf0.slice/crio-5650e15943ba05b59daf544d7e65d4c20086ee3633a1dc9d27391346307a8f6e WatchSource:0}: Error finding container 5650e15943ba05b59daf544d7e65d4c20086ee3633a1dc9d27391346307a8f6e: Status 404 returned error can't find the container with id 5650e15943ba05b59daf544d7e65d4c20086ee3633a1dc9d27391346307a8f6e Jul 31 00:41:46.641815 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:46.641729 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" event={"ID":"92f81631-6f58-4b80-8923-b2568edaacf0","Type":"ContainerStarted","Data":"5650e15943ba05b59daf544d7e65d4c20086ee3633a1dc9d27391346307a8f6e"} Jul 31 00:41:47.649278 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:47.649175 2575 generic.go:356] "Generic (PLEG): container finished" podID="92f81631-6f58-4b80-8923-b2568edaacf0" containerID="a11409dec127efd7f2d8d22c81064db5218a22d6d60c95f15ebe971b5ce13021" exitCode=0 Jul 31 00:41:47.649278 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:47.649222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" event={"ID":"92f81631-6f58-4b80-8923-b2568edaacf0","Type":"ContainerDied","Data":"a11409dec127efd7f2d8d22c81064db5218a22d6d60c95f15ebe971b5ce13021"} Jul 31 00:41:48.794036 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:48.794003 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" Jul 31 00:41:48.892580 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:48.892545 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f81631-6f58-4b80-8923-b2568edaacf0-maas-api-service-ca\") pod \"92f81631-6f58-4b80-8923-b2568edaacf0\" (UID: \"92f81631-6f58-4b80-8923-b2568edaacf0\") " Jul 31 00:41:48.892921 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:48.892896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f81631-6f58-4b80-8923-b2568edaacf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92f81631-6f58-4b80-8923-b2568edaacf0" (UID: "92f81631-6f58-4b80-8923-b2568edaacf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:41:48.993943 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:48.993849 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f81631-6f58-4b80-8923-b2568edaacf0-maas-api-service-ca\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:41:49.660845 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:49.660819 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" Jul 31 00:41:49.661139 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:49.660853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp" event={"ID":"92f81631-6f58-4b80-8923-b2568edaacf0","Type":"ContainerDied","Data":"5650e15943ba05b59daf544d7e65d4c20086ee3633a1dc9d27391346307a8f6e"} Jul 31 00:41:49.661139 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:49.660908 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5650e15943ba05b59daf544d7e65d4c20086ee3633a1dc9d27391346307a8f6e" Jul 31 00:41:51.862955 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:51.862915 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:51.962991 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:51.962960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:52.033190 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:52.033155 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp"] Jul 31 00:41:52.035632 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:52.035595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:52.036422 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:52.036398 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eee00287-m6ltp"] Jul 31 00:41:52.088376 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:52.088346 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:53.048919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.048886 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92f81631-6f58-4b80-8923-b2568edaacf0" path="/var/lib/kubelet/pods/92f81631-6f58-4b80-8923-b2568edaacf0/volumes" Jul 31 00:41:53.221308 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.221274 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:53.449821 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.449783 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:53.732768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.732681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:53.953727 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.953688 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q"] Jul 31 00:41:53.954064 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.954050 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f81631-6f58-4b80-8923-b2568edaacf0" containerName="revoke-keys" Jul 31 00:41:53.954064 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.954064 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f81631-6f58-4b80-8923-b2568edaacf0" containerName="revoke-keys" Jul 31 00:41:53.954240 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.954079 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" Jul 31 00:41:53.954240 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.954085 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="719c2015-2340-4d6d-9894-d73489452454" containerName="istio-proxy" Jul 31 00:41:53.954240 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.954179 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f81631-6f58-4b80-8923-b2568edaacf0" containerName="revoke-keys" Jul 31 00:41:53.957878 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.957849 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:53.960537 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.960513 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-eeb90a6e-openshift-default-dockercfg-w8tq6\"" Jul 31 00:41:53.968818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.968791 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q"] Jul 31 00:41:53.976654 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:53.976619 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:54.038207 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-socket\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.038207 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-data\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.038207 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19bd63a-d6ef-46df-b921-cd21217666dd-istiod-ca-cert\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.038426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-envoy\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.038426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-podinfo\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.038426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zcd\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-kube-api-access-25zcd\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.038426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-certs\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.038426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-token\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.038426 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.038420 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-credential-socket\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.139319 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-certs\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.139811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-token\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.139811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-credential-socket\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.139811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-socket\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.139811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-data\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.139811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-certs\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.139811 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139786 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-credential-socket\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.140181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19bd63a-d6ef-46df-b921-cd21217666dd-istiod-ca-cert\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.140181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-envoy\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.140181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-podinfo\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.140181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-socket\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.140181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zcd\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-kube-api-access-25zcd\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.140181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.139991 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-data\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.140505 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.140484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19bd63a-d6ef-46df-b921-cd21217666dd-istiod-ca-cert\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.142217 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.142195 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-envoy\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.142430 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.142414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-podinfo\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.147478 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.147440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-token\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.147762 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.147742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zcd\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-kube-api-access-25zcd\") pod \"e2e-derive-eeb90a6e-openshift-default-b576859b-4784q\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.270736 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.270690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:54.285476 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.285444 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:54.419480 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.419449 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q"] Jul 31 00:41:54.421304 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:41:54.421273 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19bd63a_d6ef_46df_b921_cd21217666dd.slice/crio-e2a063fe227797e811c447d302d8aaf3982e1ddc2cf87405469d9f978f248478 WatchSource:0}: Error finding container e2a063fe227797e811c447d302d8aaf3982e1ddc2cf87405469d9f978f248478: Status 404 returned error can't find the container with id e2a063fe227797e811c447d302d8aaf3982e1ddc2cf87405469d9f978f248478 Jul 31 00:41:54.423434 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.423402 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:41:54.423526 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.423469 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:41:54.423526 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.423494 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:41:54.690769 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.690737 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" event={"ID":"f19bd63a-d6ef-46df-b921-cd21217666dd","Type":"ContainerStarted","Data":"7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9"} Jul 31 00:41:54.690944 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.690787 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" event={"ID":"f19bd63a-d6ef-46df-b921-cd21217666dd","Type":"ContainerStarted","Data":"e2a063fe227797e811c447d302d8aaf3982e1ddc2cf87405469d9f978f248478"} Jul 31 00:41:54.711232 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.711181 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" podStartSLOduration=1.71116292 podStartE2EDuration="1.71116292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:41:54.708512704 +0000 UTC m=+2124.231480500" watchObservedRunningTime="2026-07-31 00:41:54.71116292 +0000 UTC m=+2124.234130664" Jul 31 00:41:54.883595 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:54.883555 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:41:55.271810 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:55.271774 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:55.276973 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:55.276944 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:55.695983 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:55.695952 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:55.697051 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:55.697029 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:41:56.044181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:56.044085 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:56.144501 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:56.144452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:57.171913 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:57.171877 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:57.521401 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:57.521301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:57.688183 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:57.688139 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:41:57.758281 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:41:57.758234 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:04.690935 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:04.690895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:42:09.779230 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:09.779196 2575 generic.go:356] "Generic (PLEG): container finished" podID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerID="7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9" exitCode=0 Jul 31 00:42:09.779645 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:09.779271 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" event={"ID":"f19bd63a-d6ef-46df-b921-cd21217666dd","Type":"ContainerDied","Data":"7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9"} Jul 31 00:42:09.779689 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:09.779666 2575 scope.go:117] "RemoveContainer" containerID="7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9" Jul 31 00:42:09.780674 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:09.780649 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:42:09.780740 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:09.780717 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:42:09.780777 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:09.780750 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:42:10.696147 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:10.696098 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:42:10.787026 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:10.786985 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" event={"ID":"f19bd63a-d6ef-46df-b921-cd21217666dd","Type":"ContainerStarted","Data":"5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70"} Jul 31 00:42:11.271346 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:11.271309 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:42:11.272591 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:11.272563 2575 patch_prober.go:28] interesting pod/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 00:42:11.272733 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:11.272613 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 00:42:12.271961 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:12.271923 2575 patch_prober.go:28] interesting pod/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 00:42:12.272422 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:12.271999 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 00:42:13.272030 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:13.271988 2575 patch_prober.go:28] interesting pod/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 00:42:13.272466 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:13.272046 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 00:42:14.271816 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:14.271769 2575 patch_prober.go:28] interesting pod/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 00:42:14.272038 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:14.271844 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 00:42:15.271228 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:15.271192 2575 patch_prober.go:28] interesting pod/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 31 00:42:15.271710 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:15.271265 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 31 00:42:16.276383 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:16.276351 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:42:16.276754 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:16.276621 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:42:16.277427 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:16.277411 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:42:27.626375 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:27.626278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:27.693035 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:27.692980 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:27.752934 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:27.752896 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:27.808063 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:27.808033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:27.913898 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:27.913862 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q"] Jul 31 00:42:27.914109 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:27.914081 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" containerID="cri-o://5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70" gracePeriod=30 Jul 31 00:42:27.928775 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:27.928736 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:28.163417 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:28.163383 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:28.638279 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:28.638240 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:28.737323 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:28.737290 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:28.840130 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:28.840095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:30.391642 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:30.391605 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:42:31.279134 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:31.279089 2575 patch_prober.go:28] interesting pod/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 00:42:31.279330 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:31.279192 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 00:42:33.168887 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.168858 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:42:33.208201 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208163 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-socket\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208201 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208205 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-envoy\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208226 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-token\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208247 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-podinfo\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208309 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19bd63a-d6ef-46df-b921-cd21217666dd-istiod-ca-cert\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208326 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-credential-socket\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208386 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-data\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208409 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-certs\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208459 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208435 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25zcd\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-kube-api-access-25zcd\") pod \"f19bd63a-d6ef-46df-b921-cd21217666dd\" (UID: \"f19bd63a-d6ef-46df-b921-cd21217666dd\") " Jul 31 00:42:33.208829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208530 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:42:33.208829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208685 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:42:33.208829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208696 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:42:33.208829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208782 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:42:33.208829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.208787 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19bd63a-d6ef-46df-b921-cd21217666dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:42:33.209167 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.209143 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.209277 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.209170 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19bd63a-d6ef-46df-b921-cd21217666dd-istiod-ca-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.209277 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.209186 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-credential-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.209277 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.209199 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-data\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.209277 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.209211 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-workload-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.210660 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.210634 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:42:33.210903 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.210880 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 00:42:33.211009 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.210931 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:42:33.211009 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.210947 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-kube-api-access-25zcd" (OuterVolumeSpecName: "kube-api-access-25zcd") pod "f19bd63a-d6ef-46df-b921-cd21217666dd" (UID: "f19bd63a-d6ef-46df-b921-cd21217666dd"). InnerVolumeSpecName "kube-api-access-25zcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:42:33.310369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.310287 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-envoy\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.310369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.310314 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-token\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.310369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.310323 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19bd63a-d6ef-46df-b921-cd21217666dd-istio-podinfo\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.310369 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.310331 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25zcd\" (UniqueName: \"kubernetes.io/projected/f19bd63a-d6ef-46df-b921-cd21217666dd-kube-api-access-25zcd\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:42:33.900532 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.900494 2575 generic.go:356] "Generic (PLEG): container finished" podID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerID="5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70" exitCode=0 Jul 31 00:42:33.900744 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.900552 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" Jul 31 00:42:33.900744 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.900579 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" event={"ID":"f19bd63a-d6ef-46df-b921-cd21217666dd","Type":"ContainerDied","Data":"5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70"} Jul 31 00:42:33.900744 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.900649 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q" event={"ID":"f19bd63a-d6ef-46df-b921-cd21217666dd","Type":"ContainerDied","Data":"e2a063fe227797e811c447d302d8aaf3982e1ddc2cf87405469d9f978f248478"} Jul 31 00:42:33.900744 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.900680 2575 scope.go:117] "RemoveContainer" containerID="5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70" Jul 31 00:42:33.914283 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.914259 2575 scope.go:117] "RemoveContainer" containerID="7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9" Jul 31 00:42:33.922778 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.922754 2575 scope.go:117] "RemoveContainer" containerID="5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70" Jul 31 00:42:33.923065 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:42:33.923047 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70\": container with ID starting with 5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70 not found: ID does not exist" containerID="5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70" Jul 31 00:42:33.923133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.923073 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70"} err="failed to get container status \"5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70\": rpc error: code = NotFound desc = could not find container \"5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70\": container with ID starting with 5087b6d68d68816809d6d73c42f4e4d3cdff0d54a20442fac57db3ddcb713e70 not found: ID does not exist" Jul 31 00:42:33.923133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.923092 2575 scope.go:117] "RemoveContainer" containerID="7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9" Jul 31 00:42:33.923435 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:42:33.923411 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9\": container with ID starting with 7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9 not found: ID does not exist" containerID="7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9" Jul 31 00:42:33.923504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.923443 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9"} err="failed to get container status \"7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9\": rpc error: code = NotFound desc = could not find container \"7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9\": container with ID starting with 7276008eb4d6cc8e0e9d7f10f34ea9b1a2392e8201ed8b134aa1fe98a31d00b9 not found: ID does not exist" Jul 31 00:42:33.925562 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.925536 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q"] Jul 31 00:42:33.930524 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:33.930483 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-eeb90a6e-openshift-default-b576859b-4784q"] Jul 31 00:42:35.049294 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:35.049249 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" path="/var/lib/kubelet/pods/f19bd63a-d6ef-46df-b921-cd21217666dd/volumes" Jul 31 00:42:40.538636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:40.538601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:43.379910 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:43.379872 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:42:46.756917 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:46.756881 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:46.862863 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:46.862828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:46.957394 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:46.957358 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:47.058028 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:47.057938 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:42:47.515952 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:42:47.515920 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:13.593475 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:13.593435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:19.414871 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:19.414834 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:19.516031 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:19.515998 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:19.614850 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:19.614812 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:35.536880 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:35.536843 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:41.114156 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:41.114098 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:41.171469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:41.171435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:41.254808 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:41.254768 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:41.354942 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:41.354889 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:43:51.884042 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:43:51.884008 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:02.950402 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:02.950323 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:08.913731 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:08.913690 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:09.018199 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:09.018164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:09.116254 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:09.116222 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:09.214058 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:09.213973 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:09.879203 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:09.879164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:09.976325 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:09.976295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:10.076928 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:10.076894 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:10.780078 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:10.780043 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:44:11.683322 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:11.683288 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:16.282642 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:16.282599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:44:17.708833 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:17.708799 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:17.807475 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:17.807445 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:18.373168 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:18.373111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:24.321406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:24.321370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:24.422697 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:24.422658 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:24.565829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:24.565786 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:24.871736 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:24.871706 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:24.897344 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:24.897309 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:44:25.023153 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:25.023099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:27.913376 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:27.913340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:28.012241 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:28.012208 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:29.033986 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:29.033947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:29.193883 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:29.193848 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:29.633288 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:29.633224 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:29.729707 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:29.729670 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:31.781105 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:31.781060 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:44:34.151939 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:34.151903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:34.199341 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:34.199302 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:52.849148 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:52.849071 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:53.546336 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:53.546265 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:53.650857 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:53.650826 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:54.018805 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:54.018769 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:59.821690 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:59.821655 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:44:59.921921 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:44:59.921888 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:00.133304 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.133211 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f"] Jul 31 00:45:00.133591 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.133576 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" Jul 31 00:45:00.133636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.133593 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" Jul 31 00:45:00.133636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.133613 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" Jul 31 00:45:00.133636 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.133619 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" Jul 31 00:45:00.133734 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.133684 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" Jul 31 00:45:00.133734 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.133693 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19bd63a-d6ef-46df-b921-cd21217666dd" containerName="istio-proxy" Jul 31 00:45:00.136649 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.136630 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8"] Jul 31 00:45:00.136768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.136748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f" Jul 31 00:45:00.139159 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.139127 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gpvxj\"" Jul 31 00:45:00.139789 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.139765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8" Jul 31 00:45:00.142458 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.142434 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8"] Jul 31 00:45:00.145931 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.145908 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f"] Jul 31 00:45:00.149930 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.149906 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f" Jul 31 00:45:00.156695 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.156664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8" Jul 31 00:45:00.298817 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.298778 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f"] Jul 31 00:45:00.300600 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:45:00.300573 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707cc972_bd3d_4645_8489_0adf74f6153b.slice/crio-b348750bc62e063a2e98c45b4ee4135e215579178e0c1d40c943e017235b1463 WatchSource:0}: Error finding container b348750bc62e063a2e98c45b4ee4135e215579178e0c1d40c943e017235b1463: Status 404 returned error can't find the container with id b348750bc62e063a2e98c45b4ee4135e215579178e0c1d40c943e017235b1463 Jul 31 00:45:00.326602 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.326464 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8"] Jul 31 00:45:00.329219 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:45:00.329193 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061304a2_259a_40f6_978e_4b73a303eef7.slice/crio-2849883e6e49334e04299f82b9907e7933f1cd5d20cbf535b7d34dbffdb993a2 WatchSource:0}: Error finding container 2849883e6e49334e04299f82b9907e7933f1cd5d20cbf535b7d34dbffdb993a2: Status 404 returned error can't find the container with id 2849883e6e49334e04299f82b9907e7933f1cd5d20cbf535b7d34dbffdb993a2 Jul 31 00:45:00.632364 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.632322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f" event={"ID":"707cc972-bd3d-4645-8489-0adf74f6153b","Type":"ContainerStarted","Data":"b348750bc62e063a2e98c45b4ee4135e215579178e0c1d40c943e017235b1463"} Jul 31 00:45:00.633822 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:00.633790 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8" event={"ID":"061304a2-259a-40f6-978e-4b73a303eef7","Type":"ContainerStarted","Data":"2849883e6e49334e04299f82b9907e7933f1cd5d20cbf535b7d34dbffdb993a2"} Jul 31 00:45:01.640187 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:01.640140 2575 generic.go:356] "Generic (PLEG): container finished" podID="707cc972-bd3d-4645-8489-0adf74f6153b" containerID="c63290d5d04999c9627f116338745503c5fe51b1c039216d364bb78f63565997" exitCode=0 Jul 31 00:45:01.640662 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:01.640229 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f" event={"ID":"707cc972-bd3d-4645-8489-0adf74f6153b","Type":"ContainerDied","Data":"c63290d5d04999c9627f116338745503c5fe51b1c039216d364bb78f63565997"} Jul 31 00:45:01.641809 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:01.641778 2575 generic.go:356] "Generic (PLEG): container finished" podID="061304a2-259a-40f6-978e-4b73a303eef7" containerID="974329c563c8e05f05463acfde832d0e07bc705778cba9a80e3833efef2bb092" exitCode=0 Jul 31 00:45:01.641935 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:01.641816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8" event={"ID":"061304a2-259a-40f6-978e-4b73a303eef7","Type":"ContainerDied","Data":"974329c563c8e05f05463acfde832d0e07bc705778cba9a80e3833efef2bb092"} Jul 31 00:45:02.790526 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:02.790489 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8" Jul 31 00:45:02.793472 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:02.793448 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f" Jul 31 00:45:03.652684 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:03.652652 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f" Jul 31 00:45:03.652873 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:03.652685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f" event={"ID":"707cc972-bd3d-4645-8489-0adf74f6153b","Type":"ContainerDied","Data":"b348750bc62e063a2e98c45b4ee4135e215579178e0c1d40c943e017235b1463"} Jul 31 00:45:03.652873 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:03.652736 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b348750bc62e063a2e98c45b4ee4135e215579178e0c1d40c943e017235b1463" Jul 31 00:45:03.654059 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:03.654034 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8" Jul 31 00:45:03.654188 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:03.654056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757645-dpzw8" event={"ID":"061304a2-259a-40f6-978e-4b73a303eef7","Type":"ContainerDied","Data":"2849883e6e49334e04299f82b9907e7933f1cd5d20cbf535b7d34dbffdb993a2"} Jul 31 00:45:03.654188 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:03.654105 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2849883e6e49334e04299f82b9907e7933f1cd5d20cbf535b7d34dbffdb993a2" Jul 31 00:45:30.714395 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:30.714263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:30.714963 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:30.714882 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f"] Jul 31 00:45:30.716519 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:30.716499 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-632e9fcf-29757645-ps54f"] Jul 31 00:45:30.778727 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:30.778692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:30.835529 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:30.835489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:30.892291 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:30.892260 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:31.049576 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:31.049503 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="707cc972-bd3d-4645-8489-0adf74f6153b" path="/var/lib/kubelet/pods/707cc972-bd3d-4645-8489-0adf74f6153b/volumes" Jul 31 00:45:31.263584 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:31.263546 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:32.874919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:32.874877 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:45:37.208328 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:37.208294 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:38.840143 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:38.840092 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:40.466757 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:40.466721 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:40.519273 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:40.519245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:43.664379 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:43.664340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:43.768619 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:43.768589 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:43.864325 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:43.864281 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:44.190844 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:44.190795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:49.633913 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:49.633880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:45:49.706147 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:45:49.706082 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:46:09.080033 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:09.079991 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:46:18.144903 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.144865 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g"] Jul 31 00:46:18.145453 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.145074 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" podUID="2f1e0077-19ab-4795-a123-2e0f34e4e93a" containerName="manager" containerID="cri-o://fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5" gracePeriod=10 Jul 31 00:46:18.413812 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.413783 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:46:18.517476 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.517441 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkzqx\" (UniqueName: \"kubernetes.io/projected/2f1e0077-19ab-4795-a123-2e0f34e4e93a-kube-api-access-nkzqx\") pod \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\" (UID: \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\") " Jul 31 00:46:18.517683 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.517543 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f1e0077-19ab-4795-a123-2e0f34e4e93a-extensions-socket-volume\") pod \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\" (UID: \"2f1e0077-19ab-4795-a123-2e0f34e4e93a\") " Jul 31 00:46:18.517947 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.517916 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1e0077-19ab-4795-a123-2e0f34e4e93a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2f1e0077-19ab-4795-a123-2e0f34e4e93a" (UID: "2f1e0077-19ab-4795-a123-2e0f34e4e93a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:46:18.519600 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.519574 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1e0077-19ab-4795-a123-2e0f34e4e93a-kube-api-access-nkzqx" (OuterVolumeSpecName: "kube-api-access-nkzqx") pod "2f1e0077-19ab-4795-a123-2e0f34e4e93a" (UID: "2f1e0077-19ab-4795-a123-2e0f34e4e93a"). InnerVolumeSpecName "kube-api-access-nkzqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:46:18.618894 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.618854 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f1e0077-19ab-4795-a123-2e0f34e4e93a-extensions-socket-volume\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:46:18.618894 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:18.618888 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkzqx\" (UniqueName: \"kubernetes.io/projected/2f1e0077-19ab-4795-a123-2e0f34e4e93a-kube-api-access-nkzqx\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:46:19.028829 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.028796 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f1e0077-19ab-4795-a123-2e0f34e4e93a" containerID="fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5" exitCode=0 Jul 31 00:46:19.029029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.028840 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" Jul 31 00:46:19.029029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.028880 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" event={"ID":"2f1e0077-19ab-4795-a123-2e0f34e4e93a","Type":"ContainerDied","Data":"fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5"} Jul 31 00:46:19.029029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.028936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g" event={"ID":"2f1e0077-19ab-4795-a123-2e0f34e4e93a","Type":"ContainerDied","Data":"0e6c163a0afe7faf4301b5c24aff97aa285e758f0855696c9642fd9edfce421e"} Jul 31 00:46:19.029029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.028962 2575 scope.go:117] "RemoveContainer" containerID="fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5" Jul 31 00:46:19.039671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.039652 2575 scope.go:117] "RemoveContainer" containerID="fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5" Jul 31 00:46:19.039948 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:46:19.039929 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5\": container with ID starting with fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5 not found: ID does not exist" containerID="fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5" Jul 31 00:46:19.039998 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.039963 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5"} err="failed to get container status \"fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5\": rpc error: code = NotFound desc = could not find container \"fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5\": container with ID starting with fe2c8ef220316c365db4fad932a04bff378c45934c7dd436e04874687d85e0d5 not found: ID does not exist" Jul 31 00:46:19.050521 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.050495 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g"] Jul 31 00:46:19.057775 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:19.057746 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-qrj2g"] Jul 31 00:46:21.048594 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:21.048555 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1e0077-19ab-4795-a123-2e0f34e4e93a" path="/var/lib/kubelet/pods/2f1e0077-19ab-4795-a123-2e0f34e4e93a/volumes" Jul 31 00:46:31.199390 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:46:31.199358 2575 scope.go:117] "RemoveContainer" containerID="ebd6a3cb42b6996fa743919d8a596eddc7104d4123b3387e2de8e621a17460ff" Jul 31 00:47:22.707666 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:22.707628 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:33.777378 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.777333 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q"] Jul 31 00:47:33.777897 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.777878 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1e0077-19ab-4795-a123-2e0f34e4e93a" containerName="manager" Jul 31 00:47:33.777949 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.777900 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1e0077-19ab-4795-a123-2e0f34e4e93a" containerName="manager" Jul 31 00:47:33.777949 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.777916 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061304a2-259a-40f6-978e-4b73a303eef7" containerName="cleanup" Jul 31 00:47:33.777949 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.777925 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="061304a2-259a-40f6-978e-4b73a303eef7" containerName="cleanup" Jul 31 00:47:33.777949 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.777941 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707cc972-bd3d-4645-8489-0adf74f6153b" containerName="cleanup" Jul 31 00:47:33.778073 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.777951 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="707cc972-bd3d-4645-8489-0adf74f6153b" containerName="cleanup" Jul 31 00:47:33.778073 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.778041 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="061304a2-259a-40f6-978e-4b73a303eef7" containerName="cleanup" Jul 31 00:47:33.778073 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.778055 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1e0077-19ab-4795-a123-2e0f34e4e93a" containerName="manager" Jul 31 00:47:33.778073 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.778066 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="707cc972-bd3d-4645-8489-0adf74f6153b" containerName="cleanup" Jul 31 00:47:33.781385 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.781366 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:33.784040 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.784018 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r29qw\"" Jul 31 00:47:33.791235 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.791209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q"] Jul 31 00:47:33.907258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.907216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ea8e0db-38b6-4151-b43d-751c831cecd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-fx69q\" (UID: \"7ea8e0db-38b6-4151-b43d-751c831cecd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:33.907468 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:33.907277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwqs\" (UniqueName: \"kubernetes.io/projected/7ea8e0db-38b6-4151-b43d-751c831cecd6-kube-api-access-gtwqs\") pod \"kuadrant-operator-controller-manager-68cf589589-fx69q\" (UID: \"7ea8e0db-38b6-4151-b43d-751c831cecd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:34.007853 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.007808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ea8e0db-38b6-4151-b43d-751c831cecd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-fx69q\" (UID: \"7ea8e0db-38b6-4151-b43d-751c831cecd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:34.008038 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.007867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwqs\" (UniqueName: \"kubernetes.io/projected/7ea8e0db-38b6-4151-b43d-751c831cecd6-kube-api-access-gtwqs\") pod \"kuadrant-operator-controller-manager-68cf589589-fx69q\" (UID: \"7ea8e0db-38b6-4151-b43d-751c831cecd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:34.008249 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.008226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ea8e0db-38b6-4151-b43d-751c831cecd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-fx69q\" (UID: \"7ea8e0db-38b6-4151-b43d-751c831cecd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:34.016938 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.016911 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwqs\" (UniqueName: \"kubernetes.io/projected/7ea8e0db-38b6-4151-b43d-751c831cecd6-kube-api-access-gtwqs\") pod \"kuadrant-operator-controller-manager-68cf589589-fx69q\" (UID: \"7ea8e0db-38b6-4151-b43d-751c831cecd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:34.092956 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.092853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:34.239445 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.239415 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q"] Jul 31 00:47:34.240684 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:47:34.240658 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea8e0db_38b6_4151_b43d_751c831cecd6.slice/crio-643cf74a3d22ba359e3f73b3af315853933b77a847917beec210976be8add8f2 WatchSource:0}: Error finding container 643cf74a3d22ba359e3f73b3af315853933b77a847917beec210976be8add8f2: Status 404 returned error can't find the container with id 643cf74a3d22ba359e3f73b3af315853933b77a847917beec210976be8add8f2 Jul 31 00:47:34.402886 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.402855 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" event={"ID":"7ea8e0db-38b6-4151-b43d-751c831cecd6","Type":"ContainerStarted","Data":"1d2ca25670b11729fd94e7aa6aaa5cf9ff73f3900da04881611e2f9337be39e6"} Jul 31 00:47:34.403081 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.402906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" event={"ID":"7ea8e0db-38b6-4151-b43d-751c831cecd6","Type":"ContainerStarted","Data":"643cf74a3d22ba359e3f73b3af315853933b77a847917beec210976be8add8f2"} Jul 31 00:47:34.403081 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.402989 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:34.422273 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:34.422219 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" podStartSLOduration=1.422202053 podStartE2EDuration="1.422202053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:47:34.420677146 +0000 UTC m=+2463.943644930" watchObservedRunningTime="2026-07-31 00:47:34.422202053 +0000 UTC m=+2463.945169797" Jul 31 00:47:45.410060 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:45.410028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fx69q" Jul 31 00:47:51.962179 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:51.962141 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:52.281178 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:52.281071 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:52.623905 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:52.623819 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:55.012032 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:55.011992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:55.111792 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:55.111759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:55.188259 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:55.188226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:47:56.142174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:56.142112 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:56.453784 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:56.453746 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:56.741147 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:56.741028 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:47:56.840390 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:47:56.840345 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:48:03.881878 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:03.881843 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:48:20.062578 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:20.062524 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:48:21.807333 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:21.807276 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:48:24.180720 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:24.180635 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:48:31.231598 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:31.231564 2575 scope.go:117] "RemoveContainer" containerID="a11409dec127efd7f2d8d22c81064db5218a22d6d60c95f15ebe971b5ce13021" Jul 31 00:48:40.370783 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:40.370744 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:48:40.422895 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:40.422861 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:48:43.975262 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:43.975224 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:48:54.602413 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.602378 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v"] Jul 31 00:48:54.607083 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.607043 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" Jul 31 00:48:54.610409 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.610375 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gpvxj\"" Jul 31 00:48:54.610571 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.610528 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 00:48:54.614539 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.614510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v"] Jul 31 00:48:54.640704 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.640669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a49cd5-e807-4950-aa3a-e0afbd8190a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v\" (UID: \"32a49cd5-e807-4950-aa3a-e0afbd8190a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" Jul 31 00:48:54.742185 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.742140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a49cd5-e807-4950-aa3a-e0afbd8190a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v\" (UID: \"32a49cd5-e807-4950-aa3a-e0afbd8190a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" Jul 31 00:48:54.742764 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.742740 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a49cd5-e807-4950-aa3a-e0afbd8190a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v\" (UID: \"32a49cd5-e807-4950-aa3a-e0afbd8190a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" Jul 31 00:48:54.922492 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:54.922449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" Jul 31 00:48:55.060602 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:55.060507 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v"] Jul 31 00:48:55.062692 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:48:55.062666 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a49cd5_e807_4950_aa3a_e0afbd8190a6.slice/crio-2c4429de7b5e4deef7e708e07d50cbb2d97af00b8c33cd7cbfda258475b384ad WatchSource:0}: Error finding container 2c4429de7b5e4deef7e708e07d50cbb2d97af00b8c33cd7cbfda258475b384ad: Status 404 returned error can't find the container with id 2c4429de7b5e4deef7e708e07d50cbb2d97af00b8c33cd7cbfda258475b384ad Jul 31 00:48:55.809736 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:55.809691 2575 generic.go:356] "Generic (PLEG): container finished" podID="32a49cd5-e807-4950-aa3a-e0afbd8190a6" containerID="0c9a490571c285b1a64a924910faeba8f77a0b26bd2e85b919a874909c10192a" exitCode=0 Jul 31 00:48:55.810174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:55.809784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" event={"ID":"32a49cd5-e807-4950-aa3a-e0afbd8190a6","Type":"ContainerDied","Data":"0c9a490571c285b1a64a924910faeba8f77a0b26bd2e85b919a874909c10192a"} Jul 31 00:48:55.810174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:55.809849 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" event={"ID":"32a49cd5-e807-4950-aa3a-e0afbd8190a6","Type":"ContainerStarted","Data":"2c4429de7b5e4deef7e708e07d50cbb2d97af00b8c33cd7cbfda258475b384ad"} Jul 31 00:48:56.956919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:56.956892 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" Jul 31 00:48:57.062991 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:57.062955 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a49cd5-e807-4950-aa3a-e0afbd8190a6-maas-api-service-ca\") pod \"32a49cd5-e807-4950-aa3a-e0afbd8190a6\" (UID: \"32a49cd5-e807-4950-aa3a-e0afbd8190a6\") " Jul 31 00:48:57.063326 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:57.063302 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a49cd5-e807-4950-aa3a-e0afbd8190a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32a49cd5-e807-4950-aa3a-e0afbd8190a6" (UID: "32a49cd5-e807-4950-aa3a-e0afbd8190a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:48:57.164180 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:57.164145 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a49cd5-e807-4950-aa3a-e0afbd8190a6-maas-api-service-ca\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:48:57.821742 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:57.821712 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" Jul 31 00:48:57.821931 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:57.821760 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v" event={"ID":"32a49cd5-e807-4950-aa3a-e0afbd8190a6","Type":"ContainerDied","Data":"2c4429de7b5e4deef7e708e07d50cbb2d97af00b8c33cd7cbfda258475b384ad"} Jul 31 00:48:57.821931 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:48:57.821804 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c4429de7b5e4deef7e708e07d50cbb2d97af00b8c33cd7cbfda258475b384ad" Jul 31 00:49:07.487753 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:07.487711 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:49:10.070135 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:10.070077 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:49:10.168497 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:10.168463 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:49:10.188340 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:10.188312 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v"] Jul 31 00:49:10.190900 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:10.190873 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f92bdba-qpp9v"] Jul 31 00:49:10.268928 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:10.268899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:49:11.049066 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:11.049030 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a49cd5-e807-4950-aa3a-e0afbd8190a6" path="/var/lib/kubelet/pods/32a49cd5-e807-4950-aa3a-e0afbd8190a6/volumes" Jul 31 00:49:13.679850 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:13.679802 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:49:14.876323 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:14.876285 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:49:18.761501 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:18.761463 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:49:32.997539 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:32.997494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:49:55.195879 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:55.195800 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:49:55.623375 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:49:55.623281 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:01.268057 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:01.268013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:32.111283 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:32.111245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:32.167630 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:32.167587 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:32.394655 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:32.394573 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:32.653398 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:32.653374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:34.286455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:34.286419 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:50:44.057986 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:44.057876 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:45.345201 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:45.345165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:46.483643 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:46.483609 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:46.982260 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:46.982226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:50:52.617943 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:52.617908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:52.722000 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:52.721969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:53.319698 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:53.319652 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:53.683724 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:53.683687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:50:58.859510 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:58.859471 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:58.914876 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:58.914842 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:58.994723 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:58.994688 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:50:59.093058 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:50:59.093022 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:09.625023 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:09.624985 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:09.875483 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:09.875383 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:30.877113 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:30.876978 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:31.266070 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:31.266029 2575 scope.go:117] "RemoveContainer" containerID="c63290d5d04999c9627f116338745503c5fe51b1c039216d364bb78f63565997" Jul 31 00:51:36.861556 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:36.861501 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:36.962026 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:36.961987 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:37.225518 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:37.225487 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:37.548477 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:37.548371 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:37.947872 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:37.947840 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:40.273938 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:40.273900 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:40.339413 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:40.339368 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:41.396009 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:41.395970 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:41.451269 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:41.451211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:41.881032 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:41.880998 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:41.969065 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:41.969028 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:51:43.879145 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:51:43.879094 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:52:05.360307 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:05.360270 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:05.715321 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:05.715287 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:06.006453 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.006322 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp"] Jul 31 00:52:06.006930 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.006905 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a49cd5-e807-4950-aa3a-e0afbd8190a6" containerName="revoke-keys" Jul 31 00:52:06.006930 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.006932 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a49cd5-e807-4950-aa3a-e0afbd8190a6" containerName="revoke-keys" Jul 31 00:52:06.007155 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.007033 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a49cd5-e807-4950-aa3a-e0afbd8190a6" containerName="revoke-keys" Jul 31 00:52:06.010629 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.010607 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.013015 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.012989 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0fc38b-openshift-default-dockercfg-2kzph\"" Jul 31 00:52:06.019856 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.019830 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp"] Jul 31 00:52:06.031100 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.031064 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:06.060340 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-socket\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.060340 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istiod-ca-cert\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.060570 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-certs\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.060570 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-credential-socket\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.060570 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-envoy\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.060570 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-podinfo\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.060706 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-data\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.060706 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-token\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.060706 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.060642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjvmp\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-kube-api-access-bjvmp\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161476 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-socket\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161476 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istiod-ca-cert\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161741 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-certs\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161741 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-credential-socket\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161741 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-envoy\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161741 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-podinfo\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161950 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-data\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161950 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-token\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.161950 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjvmp\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-kube-api-access-bjvmp\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.162102 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.161989 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-credential-socket\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.162102 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.162015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-certs\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.162102 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.162067 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-socket\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.162273 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.162249 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-data\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.162315 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.162269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istiod-ca-cert\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.164436 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.164412 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-envoy\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.164768 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.164745 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-podinfo\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.171772 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.171745 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-token\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.171922 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.171901 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjvmp\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-kube-api-access-bjvmp\") pod \"e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.322635 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.322542 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:06.340783 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.340749 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:06.484466 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.484246 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp"] Jul 31 00:52:06.486811 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:52:06.486781 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7282d1_91f7_4fd3_8ed7_24833899b1f1.slice/crio-641f365abab949626485c5e9d959055ed95876f55fadc3225b9ff3a608cca908 WatchSource:0}: Error finding container 641f365abab949626485c5e9d959055ed95876f55fadc3225b9ff3a608cca908: Status 404 returned error can't find the container with id 641f365abab949626485c5e9d959055ed95876f55fadc3225b9ff3a608cca908 Jul 31 00:52:06.489444 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.489409 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:52:06.489559 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.489499 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:52:06.489559 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.489543 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:52:06.766131 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.766088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" event={"ID":"9f7282d1-91f7-4fd3-8ed7-24833899b1f1","Type":"ContainerStarted","Data":"cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576"} Jul 31 00:52:06.766337 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.766180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" event={"ID":"9f7282d1-91f7-4fd3-8ed7-24833899b1f1","Type":"ContainerStarted","Data":"641f365abab949626485c5e9d959055ed95876f55fadc3225b9ff3a608cca908"} Jul 31 00:52:06.790106 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:06.790046 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" podStartSLOduration=1.7900284549999999 podStartE2EDuration="1.790028455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:52:06.788874889 +0000 UTC m=+2736.311842659" watchObservedRunningTime="2026-07-31 00:52:06.790028455 +0000 UTC m=+2736.312996199" Jul 31 00:52:07.323318 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:07.323278 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:07.329787 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:07.329747 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:07.770888 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:07.770852 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:07.771847 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:07.771826 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:08.743612 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:08.743567 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:08.835828 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:08.835790 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:09.860575 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:09.860539 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:10.013397 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:10.013361 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:10.350289 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:10.350258 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:10.451309 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:10.451274 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:12.390300 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:12.390255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:52:22.772391 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:22.772350 2575 patch_prober.go:28] interesting pod/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 00:52:22.772837 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:22.772410 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 00:52:22.847866 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:22.847829 2575 generic.go:356] "Generic (PLEG): container finished" podID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerID="cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576" exitCode=0 Jul 31 00:52:22.848044 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:22.847886 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" event={"ID":"9f7282d1-91f7-4fd3-8ed7-24833899b1f1","Type":"ContainerDied","Data":"cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576"} Jul 31 00:52:22.848353 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:22.848337 2575 scope.go:117] "RemoveContainer" containerID="cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576" Jul 31 00:52:22.849539 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:22.849510 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:52:22.849628 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:22.849581 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:52:22.849628 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:22.849614 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:52:23.855862 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:23.855829 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" event={"ID":"9f7282d1-91f7-4fd3-8ed7-24833899b1f1","Type":"ContainerStarted","Data":"85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3"} Jul 31 00:52:24.323749 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:24.323703 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:24.324013 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:24.323989 2575 patch_prober.go:28] interesting pod/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 00:52:24.324072 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:24.324051 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 00:52:25.323518 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:25.323474 2575 patch_prober.go:28] interesting pod/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 00:52:25.323913 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:25.323542 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 00:52:26.323461 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:26.323420 2575 patch_prober.go:28] interesting pod/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 00:52:26.323650 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:26.323496 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 00:52:26.483025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:26.482989 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:26.530756 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:26.530688 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:27.323094 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:27.323054 2575 patch_prober.go:28] interesting pod/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 00:52:27.323297 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:27.323139 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 00:52:28.323719 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:28.323677 2575 patch_prober.go:28] interesting pod/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 31 00:52:28.324099 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:28.323739 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 31 00:52:29.327473 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:29.327436 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:29.327879 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:29.327697 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:29.328590 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:29.328573 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:52:33.798445 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:33.798412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:48.979976 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:48.979940 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z"] Jul 31 00:52:48.983629 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:48.983601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:48.987314 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:48.987285 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c23a2c\"/\"kube-root-ca.crt\"" Jul 31 00:52:48.987314 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:48.987308 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c23a2c\"/\"default-dockercfg-5tpjs\"" Jul 31 00:52:48.987500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:48.987337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c23a2c\"/\"test-model-c23a2c-kserve-self-signed-certs\"" Jul 31 00:52:48.987500 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:48.987343 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c23a2c\"/\"openshift-service-ca.crt\"" Jul 31 00:52:48.993004 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:48.992980 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z"] Jul 31 00:52:49.007334 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.007294 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:52:49.065901 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.065862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-home\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.066105 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.065917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a404d53-3216-4fea-89f9-9a38e64c23c1-tls-certs\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.066105 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.066029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-model-cache\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.066105 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.066060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndg9b\" (UniqueName: \"kubernetes.io/projected/1a404d53-3216-4fea-89f9-9a38e64c23c1-kube-api-access-ndg9b\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.066105 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.066094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-dshm\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.066296 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.066139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-tmp-dir\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.167213 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-home\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.167406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a404d53-3216-4fea-89f9-9a38e64c23c1-tls-certs\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.167406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167285 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-model-cache\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.167406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndg9b\" (UniqueName: \"kubernetes.io/projected/1a404d53-3216-4fea-89f9-9a38e64c23c1-kube-api-access-ndg9b\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.167406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-dshm\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.167406 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-tmp-dir\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.167666 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-home\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.167725 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167695 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-model-cache\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.168011 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.167980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-tmp-dir\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.169635 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.169610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-dshm\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.169919 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.169900 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a404d53-3216-4fea-89f9-9a38e64c23c1-tls-certs\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.174861 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.174835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndg9b\" (UniqueName: \"kubernetes.io/projected/1a404d53-3216-4fea-89f9-9a38e64c23c1-kube-api-access-ndg9b\") pod \"test-model-c23a2c-kserve-669bb78d6f-l625z\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.295400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.295295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:49.433446 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.433413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z"] Jul 31 00:52:49.435596 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:52:49.435561 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a404d53_3216_4fea_89f9_9a38e64c23c1.slice/crio-45649c782afb8ac1b2ede4b1cd5682f736ebe4855f70a8d48442411db68ee2c9 WatchSource:0}: Error finding container 45649c782afb8ac1b2ede4b1cd5682f736ebe4855f70a8d48442411db68ee2c9: Status 404 returned error can't find the container with id 45649c782afb8ac1b2ede4b1cd5682f736ebe4855f70a8d48442411db68ee2c9 Jul 31 00:52:49.989229 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.989186 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" event={"ID":"1a404d53-3216-4fea-89f9-9a38e64c23c1","Type":"ContainerStarted","Data":"fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0"} Jul 31 00:52:49.989229 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:49.989232 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" event={"ID":"1a404d53-3216-4fea-89f9-9a38e64c23c1","Type":"ContainerStarted","Data":"45649c782afb8ac1b2ede4b1cd5682f736ebe4855f70a8d48442411db68ee2c9"} Jul 31 00:52:50.005032 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:50.004979 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" podStartSLOduration=2.004961529 podStartE2EDuration="2.004961529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:52:50.003915222 +0000 UTC m=+2779.526882962" watchObservedRunningTime="2026-07-31 00:52:50.004961529 +0000 UTC m=+2779.527929309" Jul 31 00:52:52.483779 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:52.483690 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:52:59.296154 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:59.296100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:59.296558 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:59.296171 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:52:59.309469 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:52:59.309438 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:53:00.051138 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:53:00.051094 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:53:12.321479 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:53:12.321444 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:53:15.289805 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:53:15.289758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:53:27.080703 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:53:27.080660 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:53:59.904268 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:53:59.904232 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z"] Jul 31 00:53:59.923143 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:53:59.923097 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:00.956611 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:00.956569 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:03.482793 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:03.482751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:54:11.624402 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:11.624367 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:11.725720 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:11.725682 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:11.792642 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:11.792607 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:12.188400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:12.188367 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:13.889915 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:13.889876 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:54:14.924252 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:14.924188 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" podUID="1a404d53-3216-4fea-89f9-9a38e64c23c1" containerName="main" containerID="cri-o://fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0" gracePeriod=45 Jul 31 00:54:15.178899 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.178827 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:54:15.270455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270419 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a404d53-3216-4fea-89f9-9a38e64c23c1-tls-certs\") pod \"1a404d53-3216-4fea-89f9-9a38e64c23c1\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " Jul 31 00:54:15.270455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-model-cache\") pod \"1a404d53-3216-4fea-89f9-9a38e64c23c1\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " Jul 31 00:54:15.270721 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270486 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndg9b\" (UniqueName: \"kubernetes.io/projected/1a404d53-3216-4fea-89f9-9a38e64c23c1-kube-api-access-ndg9b\") pod \"1a404d53-3216-4fea-89f9-9a38e64c23c1\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " Jul 31 00:54:15.270721 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270536 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-home\") pod \"1a404d53-3216-4fea-89f9-9a38e64c23c1\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " Jul 31 00:54:15.270721 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270559 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-tmp-dir\") pod \"1a404d53-3216-4fea-89f9-9a38e64c23c1\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " Jul 31 00:54:15.270721 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270607 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-dshm\") pod \"1a404d53-3216-4fea-89f9-9a38e64c23c1\" (UID: \"1a404d53-3216-4fea-89f9-9a38e64c23c1\") " Jul 31 00:54:15.270918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270762 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a404d53-3216-4fea-89f9-9a38e64c23c1" (UID: "1a404d53-3216-4fea-89f9-9a38e64c23c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:15.270918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270811 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a404d53-3216-4fea-89f9-9a38e64c23c1" (UID: "1a404d53-3216-4fea-89f9-9a38e64c23c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:15.270918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-home" (OuterVolumeSpecName: "home") pod "1a404d53-3216-4fea-89f9-9a38e64c23c1" (UID: "1a404d53-3216-4fea-89f9-9a38e64c23c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:15.270918 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270915 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-model-cache\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:15.271065 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.270932 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-tmp-dir\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:15.272791 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.272766 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a404d53-3216-4fea-89f9-9a38e64c23c1-kube-api-access-ndg9b" (OuterVolumeSpecName: "kube-api-access-ndg9b") pod "1a404d53-3216-4fea-89f9-9a38e64c23c1" (UID: "1a404d53-3216-4fea-89f9-9a38e64c23c1"). InnerVolumeSpecName "kube-api-access-ndg9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:54:15.272791 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.272768 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-dshm" (OuterVolumeSpecName: "dshm") pod "1a404d53-3216-4fea-89f9-9a38e64c23c1" (UID: "1a404d53-3216-4fea-89f9-9a38e64c23c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:15.272954 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.272771 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a404d53-3216-4fea-89f9-9a38e64c23c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a404d53-3216-4fea-89f9-9a38e64c23c1" (UID: "1a404d53-3216-4fea-89f9-9a38e64c23c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 00:54:15.372086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.372044 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a404d53-3216-4fea-89f9-9a38e64c23c1-tls-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:15.372086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.372080 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndg9b\" (UniqueName: \"kubernetes.io/projected/1a404d53-3216-4fea-89f9-9a38e64c23c1-kube-api-access-ndg9b\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:15.372086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.372093 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-home\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:15.372361 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.372106 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a404d53-3216-4fea-89f9-9a38e64c23c1-dshm\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:15.425482 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.425448 2575 generic.go:356] "Generic (PLEG): container finished" podID="1a404d53-3216-4fea-89f9-9a38e64c23c1" containerID="fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0" exitCode=0 Jul 31 00:54:15.425679 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.425498 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" Jul 31 00:54:15.425679 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.425515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" event={"ID":"1a404d53-3216-4fea-89f9-9a38e64c23c1","Type":"ContainerDied","Data":"fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0"} Jul 31 00:54:15.425679 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.425567 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z" event={"ID":"1a404d53-3216-4fea-89f9-9a38e64c23c1","Type":"ContainerDied","Data":"45649c782afb8ac1b2ede4b1cd5682f736ebe4855f70a8d48442411db68ee2c9"} Jul 31 00:54:15.425679 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.425585 2575 scope.go:117] "RemoveContainer" containerID="fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0" Jul 31 00:54:15.435527 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.435494 2575 scope.go:117] "RemoveContainer" containerID="fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0" Jul 31 00:54:15.435858 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:54:15.435839 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0\": container with ID starting with fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0 not found: ID does not exist" containerID="fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0" Jul 31 00:54:15.435920 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.435868 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0"} err="failed to get container status \"fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0\": rpc error: code = NotFound desc = could not find container \"fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0\": container with ID starting with fad2679bd80c81fa7e54111de3fd692b566fd810c1ef22985cb0368c138a05d0 not found: ID does not exist" Jul 31 00:54:15.447436 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.447399 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z"] Jul 31 00:54:15.449425 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:15.449400 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c23a2c/test-model-c23a2c-kserve-669bb78d6f-l625z"] Jul 31 00:54:17.049916 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:17.049876 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a404d53-3216-4fea-89f9-9a38e64c23c1" path="/var/lib/kubelet/pods/1a404d53-3216-4fea-89f9-9a38e64c23c1/volumes" Jul 31 00:54:17.957025 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:17.956988 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:18.011189 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:18.011159 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:18.219720 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:18.219634 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:18.438658 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:18.438621 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:29.188586 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:29.188491 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:29.257456 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:29.257415 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:29.310549 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:29.310507 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:29.367516 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:29.367484 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:29.736082 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:29.736048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:31.381635 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:31.381599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:54:35.427035 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:35.426988 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:35.526649 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:35.526612 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:35.672594 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:35.672558 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp"] Jul 31 00:54:35.672868 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:35.672835 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" containerID="cri-o://85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3" gracePeriod=30 Jul 31 00:54:35.691749 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:35.691659 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:35.964018 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:35.963916 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:36.306405 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:36.306261 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:36.631474 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:36.631383 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:36.843379 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:36.843339 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:39.252579 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:39.252519 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:39.350543 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:39.350497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:40.520348 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:40.520313 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:40.766889 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:40.766849 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:40.947610 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:40.947578 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:54:40.964246 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:40.964215 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:41.067056 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.066974 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:54:41.114083 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114047 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-token\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114105 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-data\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114175 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-socket\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114199 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-envoy\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114215 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-podinfo\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114238 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-credential-socket\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114262 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjvmp\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-kube-api-access-bjvmp\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114632 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istiod-ca-cert\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114632 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114360 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-certs\") pod \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\" (UID: \"9f7282d1-91f7-4fd3-8ed7-24833899b1f1\") " Jul 31 00:54:41.114632 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114441 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:41.114632 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114545 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:41.114844 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114671 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:41.114844 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114758 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-data\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.114844 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114778 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.114844 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114794 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:41.115020 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.114858 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:54:41.116509 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.116484 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:54:41.116619 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.116514 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:54:41.116789 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.116770 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 00:54:41.116848 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.116820 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-kube-api-access-bjvmp" (OuterVolumeSpecName: "kube-api-access-bjvmp") pod "9f7282d1-91f7-4fd3-8ed7-24833899b1f1" (UID: "9f7282d1-91f7-4fd3-8ed7-24833899b1f1"). InnerVolumeSpecName "kube-api-access-bjvmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:54:41.216149 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.216089 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-token\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.216149 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.216147 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-envoy\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.216383 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.216161 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istio-podinfo\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.216383 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.216175 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-credential-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.216383 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.216189 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjvmp\" (UniqueName: \"kubernetes.io/projected/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-kube-api-access-bjvmp\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.216383 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.216201 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-istiod-ca-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.216383 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.216212 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7282d1-91f7-4fd3-8ed7-24833899b1f1-workload-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:54:41.557818 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.557778 2575 generic.go:356] "Generic (PLEG): container finished" podID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerID="85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3" exitCode=0 Jul 31 00:54:41.558258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.557837 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" Jul 31 00:54:41.558258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.557837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" event={"ID":"9f7282d1-91f7-4fd3-8ed7-24833899b1f1","Type":"ContainerDied","Data":"85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3"} Jul 31 00:54:41.558258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.557983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp" event={"ID":"9f7282d1-91f7-4fd3-8ed7-24833899b1f1","Type":"ContainerDied","Data":"641f365abab949626485c5e9d959055ed95876f55fadc3225b9ff3a608cca908"} Jul 31 00:54:41.558258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.558020 2575 scope.go:117] "RemoveContainer" containerID="85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3" Jul 31 00:54:41.568545 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.568525 2575 scope.go:117] "RemoveContainer" containerID="cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576" Jul 31 00:54:41.576647 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.576626 2575 scope.go:117] "RemoveContainer" containerID="85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3" Jul 31 00:54:41.576944 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:54:41.576922 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3\": container with ID starting with 85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3 not found: ID does not exist" containerID="85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3" Jul 31 00:54:41.576983 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.576955 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3"} err="failed to get container status \"85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3\": rpc error: code = NotFound desc = could not find container \"85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3\": container with ID starting with 85f972535b60adbb3f44513e36e12feb3ea3133cb034c28695a81e60f75a12d3 not found: ID does not exist" Jul 31 00:54:41.576983 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.576975 2575 scope.go:117] "RemoveContainer" containerID="cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576" Jul 31 00:54:41.577269 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:54:41.577251 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576\": container with ID starting with cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576 not found: ID does not exist" containerID="cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576" Jul 31 00:54:41.577314 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.577275 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576"} err="failed to get container status \"cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576\": rpc error: code = NotFound desc = could not find container \"cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576\": container with ID starting with cd21fdac6efbc8cc3283deb9c7d111bb1102a3d2d4351d89ee51462160d9f576 not found: ID does not exist" Jul 31 00:54:41.580238 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.580215 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp"] Jul 31 00:54:41.583872 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:41.583848 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0fc38b-openshift-default-8474c76c8d-s4rjp"] Jul 31 00:54:42.985133 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:42.985079 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:54:43.050512 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:54:43.050472 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" path="/var/lib/kubelet/pods/9f7282d1-91f7-4fd3-8ed7-24833899b1f1/volumes" Jul 31 00:55:04.297881 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:04.297842 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:04.828448 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:04.828402 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:05.058652 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.058617 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc"] Jul 31 00:55:05.059033 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059016 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" Jul 31 00:55:05.059103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059036 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" Jul 31 00:55:05.059103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059051 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" Jul 31 00:55:05.059103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059057 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" Jul 31 00:55:05.059103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059067 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a404d53-3216-4fea-89f9-9a38e64c23c1" containerName="main" Jul 31 00:55:05.059103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059073 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a404d53-3216-4fea-89f9-9a38e64c23c1" containerName="main" Jul 31 00:55:05.059377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059153 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" Jul 31 00:55:05.059377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059167 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7282d1-91f7-4fd3-8ed7-24833899b1f1" containerName="istio-proxy" Jul 31 00:55:05.059377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.059179 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a404d53-3216-4fea-89f9-9a38e64c23c1" containerName="main" Jul 31 00:55:05.064074 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.064050 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.066634 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.066614 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-53f734-openshift-default-dockercfg-dvc8q\"" Jul 31 00:55:05.073759 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.073732 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc"] Jul 31 00:55:05.084996 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.084922 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:05.127525 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-certs\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.127723 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e883194f-105a-4ee2-a3f4-48acfff70283-istiod-ca-cert\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.127723 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-socket\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.127844 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-envoy\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.127844 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-data\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.127844 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952pz\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-kube-api-access-952pz\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.128012 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e883194f-105a-4ee2-a3f4-48acfff70283-istio-podinfo\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.128012 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-credential-socket\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.128012 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.127897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-istio-token\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229247 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952pz\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-kube-api-access-952pz\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229247 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e883194f-105a-4ee2-a3f4-48acfff70283-istio-podinfo\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-credential-socket\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-istio-token\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-certs\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e883194f-105a-4ee2-a3f4-48acfff70283-istiod-ca-cert\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-socket\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229534 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-envoy\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.229846 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.229540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-data\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.230028 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.230003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-data\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.230349 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.230327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-certs\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.230653 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.230628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-credential-socket\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.230901 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.230875 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-socket\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.231056 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.230965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e883194f-105a-4ee2-a3f4-48acfff70283-istiod-ca-cert\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.233605 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.233580 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-envoy\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.233727 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.233705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e883194f-105a-4ee2-a3f4-48acfff70283-istio-podinfo\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.239078 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.239038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-istio-token\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.239258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.239100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-952pz\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-kube-api-access-952pz\") pod \"e2e-api-b-53f734-openshift-default-7898f59794-2ldlc\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.377063 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.376963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:05.527420 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.527389 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc"] Jul 31 00:55:05.529770 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:55:05.529734 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode883194f_105a_4ee2_a3f4_48acfff70283.slice/crio-6512df5221b74c4187393ce578f86fc8487f151791b025835205a4daf6312e6d WatchSource:0}: Error finding container 6512df5221b74c4187393ce578f86fc8487f151791b025835205a4daf6312e6d: Status 404 returned error can't find the container with id 6512df5221b74c4187393ce578f86fc8487f151791b025835205a4daf6312e6d Jul 31 00:55:05.532360 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.532318 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:55:05.532471 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.532406 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:55:05.532471 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.532447 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:55:05.623029 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.622999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:05.686625 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.686588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" event={"ID":"e883194f-105a-4ee2-a3f4-48acfff70283","Type":"ContainerStarted","Data":"0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177"} Jul 31 00:55:05.687103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.686647 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" event={"ID":"e883194f-105a-4ee2-a3f4-48acfff70283","Type":"ContainerStarted","Data":"6512df5221b74c4187393ce578f86fc8487f151791b025835205a4daf6312e6d"} Jul 31 00:55:05.705642 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:05.705586 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podStartSLOduration=0.705565663 podStartE2EDuration="705.565663ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:55:05.703737165 +0000 UTC m=+2915.226704964" watchObservedRunningTime="2026-07-31 00:55:05.705565663 +0000 UTC m=+2915.228533408" Jul 31 00:55:06.378160 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:06.378093 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:06.383185 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:06.383156 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:06.691679 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:06.691645 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:06.692571 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:06.692550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:07.626017 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:07.625981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:07.726151 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:07.726096 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:08.761384 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:08.761345 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:08.879287 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:08.879248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:09.238652 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:09.238621 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:09.336518 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:09.336477 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:12.379377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:12.379340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:55:21.693181 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:21.693145 2575 patch_prober.go:28] interesting pod/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 00:55:21.693662 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:21.693201 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 00:55:21.787233 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:21.787147 2575 generic.go:356] "Generic (PLEG): container finished" podID="e883194f-105a-4ee2-a3f4-48acfff70283" containerID="0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177" exitCode=0 Jul 31 00:55:21.787233 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:21.787191 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" event={"ID":"e883194f-105a-4ee2-a3f4-48acfff70283","Type":"ContainerDied","Data":"0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177"} Jul 31 00:55:21.787619 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:21.787603 2575 scope.go:117] "RemoveContainer" containerID="0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177" Jul 31 00:55:21.788657 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:21.788626 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:55:21.788751 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:21.788725 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:55:21.788808 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:21.788779 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:55:22.794956 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:22.794916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" event={"ID":"e883194f-105a-4ee2-a3f4-48acfff70283","Type":"ContainerStarted","Data":"b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f"} Jul 31 00:55:23.377506 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:23.377463 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:23.377722 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:23.377697 2575 patch_prober.go:28] interesting pod/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 00:55:23.377771 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:23.377752 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 00:55:24.378380 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:24.378338 2575 patch_prober.go:28] interesting pod/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 00:55:24.378797 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:24.378412 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 00:55:25.377963 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:25.377925 2575 patch_prober.go:28] interesting pod/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 00:55:25.378175 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:25.377992 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 00:55:26.378448 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:26.378406 2575 patch_prober.go:28] interesting pod/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 00:55:26.378960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:26.378493 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 00:55:27.378104 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:27.378067 2575 patch_prober.go:28] interesting pod/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 31 00:55:27.378331 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:27.378162 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 31 00:55:28.381996 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:28.381966 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:28.382388 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:28.382307 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:28.383002 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:28.382985 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:55:31.310445 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:31.310405 2575 scope.go:117] "RemoveContainer" containerID="0c9a490571c285b1a64a924910faeba8f77a0b26bd2e85b919a874909c10192a" Jul 31 00:55:37.782794 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:37.782757 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:40.751847 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.751809 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9"] Jul 31 00:55:40.755180 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.755155 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" Jul 31 00:55:40.757687 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.757650 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 00:55:40.757836 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.757729 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gpvxj\"" Jul 31 00:55:40.761892 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.761864 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9"] Jul 31 00:55:40.886496 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.886458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93b2f71-a307-4a5f-b50f-8864d19fcca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-943c1a-df2j9\" (UID: \"b93b2f71-a307-4a5f-b50f-8864d19fcca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" Jul 31 00:55:40.961648 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.961604 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:40.987808 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.987761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93b2f71-a307-4a5f-b50f-8864d19fcca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-943c1a-df2j9\" (UID: \"b93b2f71-a307-4a5f-b50f-8864d19fcca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" Jul 31 00:55:40.988486 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:40.988464 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93b2f71-a307-4a5f-b50f-8864d19fcca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-943c1a-df2j9\" (UID: \"b93b2f71-a307-4a5f-b50f-8864d19fcca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" Jul 31 00:55:41.068178 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:41.068073 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" Jul 31 00:55:41.204784 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:41.204758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9"] Jul 31 00:55:41.206998 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:55:41.206965 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93b2f71_a307_4a5f_b50f_8864d19fcca9.slice/crio-5a72c9dacc1a068929adca6f72609f0ad5abb8b12319a198dd842c0a49119a9d WatchSource:0}: Error finding container 5a72c9dacc1a068929adca6f72609f0ad5abb8b12319a198dd842c0a49119a9d: Status 404 returned error can't find the container with id 5a72c9dacc1a068929adca6f72609f0ad5abb8b12319a198dd842c0a49119a9d Jul 31 00:55:41.888081 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:41.887985 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" event={"ID":"b93b2f71-a307-4a5f-b50f-8864d19fcca9","Type":"ContainerStarted","Data":"86c4c8e84f185399ad58c3979be32a0197d8458151595f37fe7381a6b69decde"} Jul 31 00:55:41.888081 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:41.888042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" event={"ID":"b93b2f71-a307-4a5f-b50f-8864d19fcca9","Type":"ContainerStarted","Data":"5a72c9dacc1a068929adca6f72609f0ad5abb8b12319a198dd842c0a49119a9d"} Jul 31 00:55:41.903036 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:41.902983 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" podStartSLOduration=1.9029673 podStartE2EDuration="1.9029673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:55:41.90137304 +0000 UTC m=+2951.424340784" watchObservedRunningTime="2026-07-31 00:55:41.9029673 +0000 UTC m=+2951.425935045" Jul 31 00:55:42.894555 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:42.894517 2575 generic.go:356] "Generic (PLEG): container finished" podID="b93b2f71-a307-4a5f-b50f-8864d19fcca9" containerID="86c4c8e84f185399ad58c3979be32a0197d8458151595f37fe7381a6b69decde" exitCode=0 Jul 31 00:55:42.894987 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:42.894609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" event={"ID":"b93b2f71-a307-4a5f-b50f-8864d19fcca9","Type":"ContainerDied","Data":"86c4c8e84f185399ad58c3979be32a0197d8458151595f37fe7381a6b69decde"} Jul 31 00:55:44.041769 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:44.041737 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" Jul 31 00:55:44.114980 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:44.114949 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93b2f71-a307-4a5f-b50f-8864d19fcca9-maas-api-service-ca\") pod \"b93b2f71-a307-4a5f-b50f-8864d19fcca9\" (UID: \"b93b2f71-a307-4a5f-b50f-8864d19fcca9\") " Jul 31 00:55:44.115355 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:44.115333 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93b2f71-a307-4a5f-b50f-8864d19fcca9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b93b2f71-a307-4a5f-b50f-8864d19fcca9" (UID: "b93b2f71-a307-4a5f-b50f-8864d19fcca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:55:44.216432 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:44.216381 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93b2f71-a307-4a5f-b50f-8864d19fcca9-maas-api-service-ca\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:55:44.905719 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:44.905671 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" Jul 31 00:55:44.905719 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:44.905701 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9" event={"ID":"b93b2f71-a307-4a5f-b50f-8864d19fcca9","Type":"ContainerDied","Data":"5a72c9dacc1a068929adca6f72609f0ad5abb8b12319a198dd842c0a49119a9d"} Jul 31 00:55:44.905952 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:44.905747 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a72c9dacc1a068929adca6f72609f0ad5abb8b12319a198dd842c0a49119a9d" Jul 31 00:55:56.250603 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:56.250508 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:56.405696 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:56.405657 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9"] Jul 31 00:55:56.408405 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:56.408372 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-943c1a-df2j9"] Jul 31 00:55:57.050014 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:57.049977 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93b2f71-a307-4a5f-b50f-8864d19fcca9" path="/var/lib/kubelet/pods/b93b2f71-a307-4a5f-b50f-8864d19fcca9/volumes" Jul 31 00:55:57.637196 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:57.637154 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:55:58.586377 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:55:58.586342 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:56:02.648719 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:02.648680 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:02.749473 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:02.749433 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:02.899903 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:02.899820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:03.132595 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:03.132561 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:18.830368 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:18.830328 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:18.929486 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:18.929441 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:19.062433 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:19.062399 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:19.115172 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:19.115061 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:19.397430 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:19.397342 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:20.186522 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:20.186478 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:56:24.903340 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:24.903303 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:25.003795 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:25.003759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:25.150703 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:25.150663 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc"] Jul 31 00:56:25.150967 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:25.150939 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" containerID="cri-o://b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f" gracePeriod=30 Jul 31 00:56:25.165110 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:25.165009 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:25.414425 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:25.414389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:25.754708 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:25.754674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:26.078964 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:26.078882 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:26.337536 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:26.337457 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:28.385418 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:28.385379 2575 patch_prober.go:28] interesting pod/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 00:56:28.385806 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:28.385444 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 00:56:28.906732 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:28.906692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:29.016455 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:29.016421 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:29.998985 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:29.998955 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:30.052364 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.052325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:30.415161 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.415137 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:56:30.488571 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.488535 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:30.534735 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534638 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e883194f-105a-4ee2-a3f4-48acfff70283-istiod-ca-cert\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.534735 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534709 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-envoy\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.534962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534742 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-952pz\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-kube-api-access-952pz\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.534962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534792 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-data\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.534962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534864 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e883194f-105a-4ee2-a3f4-48acfff70283-istio-podinfo\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.534962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534897 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-credential-socket\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.534962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534926 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-certs\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.534962 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534956 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-socket\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.535290 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.534978 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-istio-token\") pod \"e883194f-105a-4ee2-a3f4-48acfff70283\" (UID: \"e883194f-105a-4ee2-a3f4-48acfff70283\") " Jul 31 00:56:30.535347 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.535303 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:56:30.535476 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.535445 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-data" (OuterVolumeSpecName: "istio-data") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:56:30.535610 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.535472 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:56:30.535610 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.535598 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e883194f-105a-4ee2-a3f4-48acfff70283-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 00:56:30.535957 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.535908 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:56:30.537849 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.537820 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-kube-api-access-952pz" (OuterVolumeSpecName: "kube-api-access-952pz") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "kube-api-access-952pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:56:30.537937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.537919 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-istio-token" (OuterVolumeSpecName: "istio-token") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 00:56:30.537937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.537925 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e883194f-105a-4ee2-a3f4-48acfff70283-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 00:56:30.538023 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.537968 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e883194f-105a-4ee2-a3f4-48acfff70283" (UID: "e883194f-105a-4ee2-a3f4-48acfff70283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 00:56:30.589756 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.589726 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:30.636130 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636077 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-data\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:30.636130 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636107 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e883194f-105a-4ee2-a3f4-48acfff70283-istio-podinfo\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:30.636352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636142 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-credential-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:30.636352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636153 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:30.636352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636162 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-workload-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:30.636352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636170 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-istio-token\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:30.636352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636178 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e883194f-105a-4ee2-a3f4-48acfff70283-istiod-ca-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:30.636352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636185 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e883194f-105a-4ee2-a3f4-48acfff70283-istio-envoy\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:30.636352 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:30.636193 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952pz\" (UniqueName: \"kubernetes.io/projected/e883194f-105a-4ee2-a3f4-48acfff70283-kube-api-access-952pz\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 00:56:31.139580 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.139541 2575 generic.go:356] "Generic (PLEG): container finished" podID="e883194f-105a-4ee2-a3f4-48acfff70283" containerID="b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f" exitCode=0 Jul 31 00:56:31.140086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.139610 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" Jul 31 00:56:31.140086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.139640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" event={"ID":"e883194f-105a-4ee2-a3f4-48acfff70283","Type":"ContainerDied","Data":"b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f"} Jul 31 00:56:31.140086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.139693 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc" event={"ID":"e883194f-105a-4ee2-a3f4-48acfff70283","Type":"ContainerDied","Data":"6512df5221b74c4187393ce578f86fc8487f151791b025835205a4daf6312e6d"} Jul 31 00:56:31.140086 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.139722 2575 scope.go:117] "RemoveContainer" containerID="b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f" Jul 31 00:56:31.151143 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.151098 2575 scope.go:117] "RemoveContainer" containerID="0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177" Jul 31 00:56:31.157139 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.157077 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc"] Jul 31 00:56:31.160877 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.160852 2575 scope.go:117] "RemoveContainer" containerID="b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f" Jul 31 00:56:31.161216 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:56:31.161185 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f\": container with ID starting with b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f not found: ID does not exist" containerID="b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f" Jul 31 00:56:31.161337 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.161229 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f"} err="failed to get container status \"b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f\": rpc error: code = NotFound desc = could not find container \"b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f\": container with ID starting with b8f520616aaa9952d1224d85da9694ce91098d8c2cf9871dc078c1721a84d16f not found: ID does not exist" Jul 31 00:56:31.161337 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.161258 2575 scope.go:117] "RemoveContainer" containerID="0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177" Jul 31 00:56:31.161553 ip-10-0-134-191 kubenswrapper[2575]: E0731 00:56:31.161534 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177\": container with ID starting with 0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177 not found: ID does not exist" containerID="0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177" Jul 31 00:56:31.161615 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.161564 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177"} err="failed to get container status \"0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177\": rpc error: code = NotFound desc = could not find container \"0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177\": container with ID starting with 0e31fa59baf22634fd9130674726c262532292c320b6dfa2dbb527c48ee69177 not found: ID does not exist" Jul 31 00:56:31.161717 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:31.161700 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-53f734-openshift-default-7898f59794-2ldlc"] Jul 31 00:56:32.874939 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:32.874905 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:56:33.049222 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:33.049191 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" path="/var/lib/kubelet/pods/e883194f-105a-4ee2-a3f4-48acfff70283/volumes" Jul 31 00:56:45.035791 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:45.035716 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:53.898258 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:53.898219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:54.238502 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.238466 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:54.517606 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.517498 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4"] Jul 31 00:56:54.518087 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.518064 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93b2f71-a307-4a5f-b50f-8864d19fcca9" containerName="revoke-keys" Jul 31 00:56:54.518170 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.518092 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93b2f71-a307-4a5f-b50f-8864d19fcca9" containerName="revoke-keys" Jul 31 00:56:54.518170 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.518108 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" Jul 31 00:56:54.518170 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.518135 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" Jul 31 00:56:54.518315 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.518272 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93b2f71-a307-4a5f-b50f-8864d19fcca9" containerName="revoke-keys" Jul 31 00:56:54.518315 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.518285 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" Jul 31 00:56:54.518315 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.518294 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" Jul 31 00:56:54.521769 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.521744 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.524289 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.524262 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b32567-openshift-default-dockercfg-lh7k8\"" Jul 31 00:56:54.533504 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.533474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4"] Jul 31 00:56:54.539977 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.539944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:54.652542 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-envoy\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.652758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istiod-ca-cert\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.652758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-credential-socket\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.652758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfcz\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-kube-api-access-pkfcz\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.652758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-podinfo\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.652758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-token\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.652758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-socket\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.652758 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-certs\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.653006 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.652770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-data\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.753700 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.753650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istiod-ca-cert\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.753917 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.753714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-credential-socket\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.753917 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.753794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfcz\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-kube-api-access-pkfcz\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.753917 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.753839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-podinfo\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.753917 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.753860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-token\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.753917 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.753897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-socket\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.754237 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.753921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-certs\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.754237 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.753947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-data\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.754237 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.754191 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-credential-socket\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.754400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.754243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-envoy\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.754400 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.754346 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-data\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.754514 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.754474 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-certs\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.754581 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.754540 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-socket\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.754645 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.754578 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istiod-ca-cert\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.756704 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.756674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-podinfo\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.756815 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.756732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-envoy\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.761573 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.761546 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-token\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.761770 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.761747 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfcz\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-kube-api-access-pkfcz\") pod \"e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.835788 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.835691 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:54.942939 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.942899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:54.986486 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.986463 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4"] Jul 31 00:56:54.988076 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:56:54.988047 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bd3c7b_e92a_47b5_a928_363eb1d9bf74.slice/crio-84b7a8ab73a103f86c975379d6da94085fe6a09b94831d909625e6e3b87871d7 WatchSource:0}: Error finding container 84b7a8ab73a103f86c975379d6da94085fe6a09b94831d909625e6e3b87871d7: Status 404 returned error can't find the container with id 84b7a8ab73a103f86c975379d6da94085fe6a09b94831d909625e6e3b87871d7 Jul 31 00:56:54.990489 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.990451 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:56:54.990578 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.990541 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:56:54.990623 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:54.990595 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:56:55.268718 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:55.268676 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" event={"ID":"10bd3c7b-e92a-47b5-a928-363eb1d9bf74","Type":"ContainerStarted","Data":"0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c"} Jul 31 00:56:55.268906 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:55.268743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" event={"ID":"10bd3c7b-e92a-47b5-a928-363eb1d9bf74","Type":"ContainerStarted","Data":"84b7a8ab73a103f86c975379d6da94085fe6a09b94831d909625e6e3b87871d7"} Jul 31 00:56:55.290058 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:55.289999 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" podStartSLOduration=1.289982412 podStartE2EDuration="1.289982412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:56:55.28758567 +0000 UTC m=+3024.810553436" watchObservedRunningTime="2026-07-31 00:56:55.289982412 +0000 UTC m=+3024.812950157" Jul 31 00:56:55.639164 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:55.639072 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:55.836034 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:55.835987 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:56.841607 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:56.841575 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:57.208599 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:57.208565 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:57.278037 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:57.278009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:57.279581 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:57.279567 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:56:57.311681 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:57.311641 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:58.332541 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:58.332498 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:58.483970 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:58.483933 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:58.928802 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:58.928761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:56:59.025866 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:56:59.025829 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:57:01.775860 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:01.775801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:57:11.356235 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:11.356195 2575 generic.go:356] "Generic (PLEG): container finished" podID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerID="0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c" exitCode=0 Jul 31 00:57:11.356671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:11.356251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" event={"ID":"10bd3c7b-e92a-47b5-a928-363eb1d9bf74","Type":"ContainerDied","Data":"0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c"} Jul 31 00:57:11.356671 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:11.356643 2575 scope.go:117] "RemoveContainer" containerID="0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c" Jul 31 00:57:11.357744 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:11.357711 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:57:11.357804 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:11.357783 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:57:11.357842 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:11.357824 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 00:57:12.278788 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:12.278751 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:57:12.363883 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:12.363849 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" event={"ID":"10bd3c7b-e92a-47b5-a928-363eb1d9bf74","Type":"ContainerStarted","Data":"c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf"} Jul 31 00:57:12.836881 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:12.836822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:57:12.837862 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:12.837818 2575 patch_prober.go:28] interesting pod/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 00:57:12.838140 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:12.838072 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 00:57:13.836254 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:13.836200 2575 patch_prober.go:28] interesting pod/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 00:57:13.836798 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:13.836277 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 00:57:14.836364 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:14.836313 2575 patch_prober.go:28] interesting pod/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 00:57:14.836897 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:14.836388 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 00:57:15.836441 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:15.836397 2575 patch_prober.go:28] interesting pod/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 00:57:15.837070 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:15.837038 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 00:57:16.845307 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:16.845261 2575 patch_prober.go:28] interesting pod/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 31 00:57:16.845924 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:16.845884 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 31 00:57:17.840503 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:17.840468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:57:17.840770 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:17.840746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:57:17.841434 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:17.841417 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 00:57:22.233868 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:22.233832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:57:48.990591 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:48.990549 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp"] Jul 31 00:57:48.991026 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:48.990950 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" Jul 31 00:57:48.991026 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:48.990961 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e883194f-105a-4ee2-a3f4-48acfff70283" containerName="istio-proxy" Jul 31 00:57:48.992937 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:48.992919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:48.997195 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:48.997166 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6a2164\"/\"auth-test-model-6a2164-kserve-self-signed-certs\"" Jul 31 00:57:48.997358 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:48.997199 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6a2164\"/\"default-dockercfg-7chb9\"" Jul 31 00:57:48.997358 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:48.997229 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6a2164\"/\"kube-root-ca.crt\"" Jul 31 00:57:48.997358 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:48.997166 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6a2164\"/\"openshift-service-ca.crt\"" Jul 31 00:57:49.001970 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.001943 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp"] Jul 31 00:57:49.065371 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.065332 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tls-certs\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.065564 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.065377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tmp-dir\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.065564 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.065402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-model-cache\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.065564 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.065425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxs9\" (UniqueName: \"kubernetes.io/projected/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-kube-api-access-lxxs9\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.065564 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.065470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-dshm\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.065759 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.065583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-home\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.084900 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.084863 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:57:49.167048 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.166999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-home\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.167293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.167077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tls-certs\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.167293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.167105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tmp-dir\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.167293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.167157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-model-cache\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.167293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.167183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxs9\" (UniqueName: \"kubernetes.io/projected/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-kube-api-access-lxxs9\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.167293 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.167231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-dshm\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.167574 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.167471 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-home\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.167574 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.167557 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tmp-dir\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.167683 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.167628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-model-cache\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.169623 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.169597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-dshm\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.169853 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.169835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tls-certs\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.175659 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.175627 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxs9\" (UniqueName: \"kubernetes.io/projected/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-kube-api-access-lxxs9\") pod \"auth-test-model-6a2164-kserve-76d6776c47-v5zpp\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.303701 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.303609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:49.447233 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.447197 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp"] Jul 31 00:57:49.449087 ip-10-0-134-191 kubenswrapper[2575]: W0731 00:57:49.449050 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3da050e_16a4_4e13_b3ac_98f0cb1a764e.slice/crio-804d9865799327b311cb6d1522aa677b7405aa47fc1771905afc20fca630242b WatchSource:0}: Error finding container 804d9865799327b311cb6d1522aa677b7405aa47fc1771905afc20fca630242b: Status 404 returned error can't find the container with id 804d9865799327b311cb6d1522aa677b7405aa47fc1771905afc20fca630242b Jul 31 00:57:49.548593 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.548560 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" event={"ID":"b3da050e-16a4-4e13-b3ac-98f0cb1a764e","Type":"ContainerStarted","Data":"823140ae12ca09e6d276e967cebb1e61b78269fc26ec89314bcd61a8849c86d6"} Jul 31 00:57:49.548779 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.548600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" event={"ID":"b3da050e-16a4-4e13-b3ac-98f0cb1a764e","Type":"ContainerStarted","Data":"804d9865799327b311cb6d1522aa677b7405aa47fc1771905afc20fca630242b"} Jul 31 00:57:49.563877 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:49.563765 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" podStartSLOduration=1.563746059 podStartE2EDuration="1.563746059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:57:49.561972437 +0000 UTC m=+3079.084940198" watchObservedRunningTime="2026-07-31 00:57:49.563746059 +0000 UTC m=+3079.086713803" Jul 31 00:57:51.620820 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:51.620783 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:57:59.303987 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:59.303946 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:59.304439 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:59.304012 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:59.316960 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:59.316933 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:57:59.609777 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:57:59.609688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 00:58:16.491231 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.491191 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56"] Jul 31 00:58:16.494565 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.494541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.497164 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.497133 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b32567\"/\"openshift-service-ca.crt\"" Jul 31 00:58:16.497320 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.497202 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b32567\"/\"kube-root-ca.crt\"" Jul 31 00:58:16.498402 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.498376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b32567\"/\"auth-test-model-b32567-kserve-self-signed-certs\"" Jul 31 00:58:16.498616 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.498596 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b32567\"/\"default-dockercfg-bdwzj\"" Jul 31 00:58:16.505550 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.505521 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56"] Jul 31 00:58:16.513609 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.513575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:58:16.528824 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.528790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-home\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.528824 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.528831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-dshm\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.529103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.528855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64e1a75-c873-48f4-91f9-1cf25c72d662-tls-certs\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.529103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.528981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-tmp-dir\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.529103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.529010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-model-cache\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.529103 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.529033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwq9\" (UniqueName: \"kubernetes.io/projected/f64e1a75-c873-48f4-91f9-1cf25c72d662-kube-api-access-6xwq9\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.630471 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.630432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-home\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.630471 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.630471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-dshm\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.630716 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.630489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64e1a75-c873-48f4-91f9-1cf25c72d662-tls-certs\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.630716 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.630548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-tmp-dir\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.630716 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.630582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-model-cache\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.630716 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.630617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwq9\" (UniqueName: \"kubernetes.io/projected/f64e1a75-c873-48f4-91f9-1cf25c72d662-kube-api-access-6xwq9\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.630938 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.630909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-home\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.631174 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.631148 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-model-cache\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.631310 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.631233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-tmp-dir\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.632950 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.632924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-dshm\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.633609 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.633586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64e1a75-c873-48f4-91f9-1cf25c72d662-tls-certs\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.639038 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.638997 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwq9\" (UniqueName: \"kubernetes.io/projected/f64e1a75-c873-48f4-91f9-1cf25c72d662-kube-api-access-6xwq9\") pod \"auth-test-model-b32567-kserve-5f8654fb7-26c56\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.807448 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.807350 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:16.950970 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:16.950942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56"] Jul 31 00:58:17.690023 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:17.689978 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" event={"ID":"f64e1a75-c873-48f4-91f9-1cf25c72d662","Type":"ContainerStarted","Data":"d83a98ebff397d326fdfc31e509feafd1f5265c86c33b43cf5381b68bd1f73d9"} Jul 31 00:58:17.690449 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:17.690031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" event={"ID":"f64e1a75-c873-48f4-91f9-1cf25c72d662","Type":"ContainerStarted","Data":"de39cb2e0b683e38cd675783224ba896f9281e7cb9e940e9db59dde89769da03"} Jul 31 00:58:17.706864 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:17.706811 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" podStartSLOduration=1.7067944069999998 podStartE2EDuration="1.706794407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 00:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 00:58:17.705399571 +0000 UTC m=+3107.228367317" watchObservedRunningTime="2026-07-31 00:58:17.706794407 +0000 UTC m=+3107.229762152" Jul 31 00:58:19.283059 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:19.283021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:58:26.807803 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:26.807761 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:26.808295 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:26.807824 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:26.820668 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:26.820640 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:27.753852 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:27.753815 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 00:58:31.550970 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:31.550934 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 00:58:43.585815 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:43.585777 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:58:58.579285 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:58:58.579198 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:59:09.690079 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:59:09.690032 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:59:25.280284 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:59:25.280243 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:59:40.882966 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:59:40.882927 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 00:59:51.682000 ip-10-0-134-191 kubenswrapper[2575]: I0731 00:59:51.681960 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:00:04.581219 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:00:04.581167 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:00:07.181953 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:00:07.181913 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:00:22.784967 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:00:22.784870 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:00:34.287230 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:00:34.287191 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:00:49.081637 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:00:49.081600 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:01:15.081716 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:01:15.081675 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:01:25.634729 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:01:25.634697 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:01:41.283577 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:01:41.283537 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:01:54.781161 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:01:54.781061 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:02:05.581220 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:05.581183 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:02:21.483779 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:21.483738 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:02:31.392561 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:31.392522 2575 scope.go:117] "RemoveContainer" containerID="86c4c8e84f185399ad58c3979be32a0197d8458151595f37fe7381a6b69decde" Jul 31 01:02:36.977546 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:36.977502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:02:40.884208 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:40.884156 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp"] Jul 31 01:02:40.937921 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:40.937881 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:02:41.888964 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:41.888923 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56"] Jul 31 01:02:42.374407 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:42.374370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:02:47.502411 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:47.502374 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:02:52.602214 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:52.602172 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:02:55.282382 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:55.282337 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:02:55.904089 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:55.904049 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" podUID="b3da050e-16a4-4e13-b3ac-98f0cb1a764e" containerName="main" containerID="cri-o://823140ae12ca09e6d276e967cebb1e61b78269fc26ec89314bcd61a8849c86d6" gracePeriod=45 Jul 31 01:02:56.093748 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.093715 2575 generic.go:356] "Generic (PLEG): container finished" podID="b3da050e-16a4-4e13-b3ac-98f0cb1a764e" containerID="823140ae12ca09e6d276e967cebb1e61b78269fc26ec89314bcd61a8849c86d6" exitCode=0 Jul 31 01:02:56.094006 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.093776 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" event={"ID":"b3da050e-16a4-4e13-b3ac-98f0cb1a764e","Type":"ContainerDied","Data":"823140ae12ca09e6d276e967cebb1e61b78269fc26ec89314bcd61a8849c86d6"} Jul 31 01:02:56.167187 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.167097 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 01:02:56.263328 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263287 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-dshm\") pod \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " Jul 31 01:02:56.263511 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263395 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tmp-dir\") pod \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " Jul 31 01:02:56.263511 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263460 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-model-cache\") pod \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " Jul 31 01:02:56.263624 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263510 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tls-certs\") pod \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " Jul 31 01:02:56.263624 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263535 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-home\") pod \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " Jul 31 01:02:56.263624 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263572 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxxs9\" (UniqueName: \"kubernetes.io/projected/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-kube-api-access-lxxs9\") pod \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\" (UID: \"b3da050e-16a4-4e13-b3ac-98f0cb1a764e\") " Jul 31 01:02:56.263787 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263658 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3da050e-16a4-4e13-b3ac-98f0cb1a764e" (UID: "b3da050e-16a4-4e13-b3ac-98f0cb1a764e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:02:56.263843 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263781 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3da050e-16a4-4e13-b3ac-98f0cb1a764e" (UID: "b3da050e-16a4-4e13-b3ac-98f0cb1a764e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:02:56.263843 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263824 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-home" (OuterVolumeSpecName: "home") pod "b3da050e-16a4-4e13-b3ac-98f0cb1a764e" (UID: "b3da050e-16a4-4e13-b3ac-98f0cb1a764e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:02:56.263954 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263937 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-home\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:56.263997 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263958 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tmp-dir\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:56.263997 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.263969 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-model-cache\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:56.265740 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.265719 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-dshm" (OuterVolumeSpecName: "dshm") pod "b3da050e-16a4-4e13-b3ac-98f0cb1a764e" (UID: "b3da050e-16a4-4e13-b3ac-98f0cb1a764e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:02:56.265826 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.265738 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3da050e-16a4-4e13-b3ac-98f0cb1a764e" (UID: "b3da050e-16a4-4e13-b3ac-98f0cb1a764e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 01:02:56.265826 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.265747 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-kube-api-access-lxxs9" (OuterVolumeSpecName: "kube-api-access-lxxs9") pod "b3da050e-16a4-4e13-b3ac-98f0cb1a764e" (UID: "b3da050e-16a4-4e13-b3ac-98f0cb1a764e"). InnerVolumeSpecName "kube-api-access-lxxs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:02:56.364714 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.364667 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-tls-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:56.364714 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.364706 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxxs9\" (UniqueName: \"kubernetes.io/projected/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-kube-api-access-lxxs9\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:56.364714 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.364719 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da050e-16a4-4e13-b3ac-98f0cb1a764e-dshm\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:56.908599 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:56.908558 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" podUID="f64e1a75-c873-48f4-91f9-1cf25c72d662" containerName="main" containerID="cri-o://d83a98ebff397d326fdfc31e509feafd1f5265c86c33b43cf5381b68bd1f73d9" gracePeriod=45 Jul 31 01:02:57.101969 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.101937 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" Jul 31 01:02:57.102174 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.101941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp" event={"ID":"b3da050e-16a4-4e13-b3ac-98f0cb1a764e","Type":"ContainerDied","Data":"804d9865799327b311cb6d1522aa677b7405aa47fc1771905afc20fca630242b"} Jul 31 01:02:57.102174 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.102144 2575 scope.go:117] "RemoveContainer" containerID="823140ae12ca09e6d276e967cebb1e61b78269fc26ec89314bcd61a8849c86d6" Jul 31 01:02:57.105165 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.105055 2575 generic.go:356] "Generic (PLEG): container finished" podID="f64e1a75-c873-48f4-91f9-1cf25c72d662" containerID="d83a98ebff397d326fdfc31e509feafd1f5265c86c33b43cf5381b68bd1f73d9" exitCode=0 Jul 31 01:02:57.105165 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.105095 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" event={"ID":"f64e1a75-c873-48f4-91f9-1cf25c72d662","Type":"ContainerDied","Data":"d83a98ebff397d326fdfc31e509feafd1f5265c86c33b43cf5381b68bd1f73d9"} Jul 31 01:02:57.118744 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.118715 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp"] Jul 31 01:02:57.121841 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.121814 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/auth-test-model-6a2164-kserve-76d6776c47-v5zpp"] Jul 31 01:02:57.161678 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.161611 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 01:02:57.273921 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.273884 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-dshm\") pod \"f64e1a75-c873-48f4-91f9-1cf25c72d662\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " Jul 31 01:02:57.273921 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.273931 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64e1a75-c873-48f4-91f9-1cf25c72d662-tls-certs\") pod \"f64e1a75-c873-48f4-91f9-1cf25c72d662\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " Jul 31 01:02:57.274250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.273967 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xwq9\" (UniqueName: \"kubernetes.io/projected/f64e1a75-c873-48f4-91f9-1cf25c72d662-kube-api-access-6xwq9\") pod \"f64e1a75-c873-48f4-91f9-1cf25c72d662\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " Jul 31 01:02:57.274250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.273996 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-home\") pod \"f64e1a75-c873-48f4-91f9-1cf25c72d662\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " Jul 31 01:02:57.274250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.274012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-tmp-dir\") pod \"f64e1a75-c873-48f4-91f9-1cf25c72d662\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " Jul 31 01:02:57.274250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.274036 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-model-cache\") pod \"f64e1a75-c873-48f4-91f9-1cf25c72d662\" (UID: \"f64e1a75-c873-48f4-91f9-1cf25c72d662\") " Jul 31 01:02:57.274473 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.274278 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f64e1a75-c873-48f4-91f9-1cf25c72d662" (UID: "f64e1a75-c873-48f4-91f9-1cf25c72d662"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:02:57.274473 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.274298 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-home" (OuterVolumeSpecName: "home") pod "f64e1a75-c873-48f4-91f9-1cf25c72d662" (UID: "f64e1a75-c873-48f4-91f9-1cf25c72d662"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:02:57.274473 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.274381 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-model-cache" (OuterVolumeSpecName: "model-cache") pod "f64e1a75-c873-48f4-91f9-1cf25c72d662" (UID: "f64e1a75-c873-48f4-91f9-1cf25c72d662"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:02:57.274473 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.274411 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-home\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:57.274473 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.274428 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-tmp-dir\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:57.276422 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.276398 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-dshm" (OuterVolumeSpecName: "dshm") pod "f64e1a75-c873-48f4-91f9-1cf25c72d662" (UID: "f64e1a75-c873-48f4-91f9-1cf25c72d662"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:02:57.276514 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.276403 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64e1a75-c873-48f4-91f9-1cf25c72d662-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f64e1a75-c873-48f4-91f9-1cf25c72d662" (UID: "f64e1a75-c873-48f4-91f9-1cf25c72d662"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 01:02:57.276514 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.276425 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64e1a75-c873-48f4-91f9-1cf25c72d662-kube-api-access-6xwq9" (OuterVolumeSpecName: "kube-api-access-6xwq9") pod "f64e1a75-c873-48f4-91f9-1cf25c72d662" (UID: "f64e1a75-c873-48f4-91f9-1cf25c72d662"). InnerVolumeSpecName "kube-api-access-6xwq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:02:57.375295 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.375253 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-model-cache\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:57.375295 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.375285 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64e1a75-c873-48f4-91f9-1cf25c72d662-dshm\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:57.375295 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.375294 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64e1a75-c873-48f4-91f9-1cf25c72d662-tls-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:57.375295 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:57.375303 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xwq9\" (UniqueName: \"kubernetes.io/projected/f64e1a75-c873-48f4-91f9-1cf25c72d662-kube-api-access-6xwq9\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:02:58.112243 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:58.112206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" event={"ID":"f64e1a75-c873-48f4-91f9-1cf25c72d662","Type":"ContainerDied","Data":"de39cb2e0b683e38cd675783224ba896f9281e7cb9e940e9db59dde89769da03"} Jul 31 01:02:58.112434 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:58.112254 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56" Jul 31 01:02:58.112434 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:58.112260 2575 scope.go:117] "RemoveContainer" containerID="d83a98ebff397d326fdfc31e509feafd1f5265c86c33b43cf5381b68bd1f73d9" Jul 31 01:02:58.134787 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:58.134752 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56"] Jul 31 01:02:58.136935 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:58.136907 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/auth-test-model-b32567-kserve-5f8654fb7-26c56"] Jul 31 01:02:59.050296 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:59.050264 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3da050e-16a4-4e13-b3ac-98f0cb1a764e" path="/var/lib/kubelet/pods/b3da050e-16a4-4e13-b3ac-98f0cb1a764e/volumes" Jul 31 01:02:59.050695 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:02:59.050686 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64e1a75-c873-48f4-91f9-1cf25c72d662" path="/var/lib/kubelet/pods/f64e1a75-c873-48f4-91f9-1cf25c72d662/volumes" Jul 31 01:03:15.085534 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.085494 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz"] Jul 31 01:03:15.086592 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.086564 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3da050e-16a4-4e13-b3ac-98f0cb1a764e" containerName="main" Jul 31 01:03:15.086592 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.086590 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3da050e-16a4-4e13-b3ac-98f0cb1a764e" containerName="main" Jul 31 01:03:15.086764 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.086632 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64e1a75-c873-48f4-91f9-1cf25c72d662" containerName="main" Jul 31 01:03:15.086764 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.086638 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64e1a75-c873-48f4-91f9-1cf25c72d662" containerName="main" Jul 31 01:03:15.086764 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.086700 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f64e1a75-c873-48f4-91f9-1cf25c72d662" containerName="main" Jul 31 01:03:15.086764 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.086711 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3da050e-16a4-4e13-b3ac-98f0cb1a764e" containerName="main" Jul 31 01:03:15.089482 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.089460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.092134 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.092092 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b32567\"/\"kube-root-ca.crt\"" Jul 31 01:03:15.093344 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.093315 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b32567\"/\"openshift-service-ca.crt\"" Jul 31 01:03:15.093461 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.093366 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b32567\"/\"default-dockercfg-bdwzj\"" Jul 31 01:03:15.093461 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.093394 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b32567\"/\"sub-test-model-b32567-kserve-self-signed-certs\"" Jul 31 01:03:15.096783 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.096753 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz"] Jul 31 01:03:15.105209 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.105170 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:03:15.138237 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.138197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tls-certs\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.138237 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.138242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc2rn\" (UniqueName: \"kubernetes.io/projected/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-kube-api-access-fc2rn\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.138457 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.138279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-model-cache\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.138457 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.138298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-dshm\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.138556 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.138534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-home\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.138611 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.138595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tmp-dir\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239208 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tls-certs\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239379 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc2rn\" (UniqueName: \"kubernetes.io/projected/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-kube-api-access-fc2rn\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239379 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-model-cache\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239379 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-dshm\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239541 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-home\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239541 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tmp-dir\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239777 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239753 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-model-cache\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239934 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tmp-dir\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.239934 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.239872 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-home\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.241751 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.241731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-dshm\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.241941 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.241922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tls-certs\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.246802 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.246779 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc2rn\" (UniqueName: \"kubernetes.io/projected/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-kube-api-access-fc2rn\") pod \"sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.402804 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.402774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:15.545637 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:15.545605 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz"] Jul 31 01:03:15.547323 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:03:15.547291 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f7ee2f2_6fdc_430b_88e2_c1047a19df31.slice/crio-6e0a7d1dceb986dd58ac90b5763dd82a271fbc0293975c09a13e6fa7acc9e8ec WatchSource:0}: Error finding container 6e0a7d1dceb986dd58ac90b5763dd82a271fbc0293975c09a13e6fa7acc9e8ec: Status 404 returned error can't find the container with id 6e0a7d1dceb986dd58ac90b5763dd82a271fbc0293975c09a13e6fa7acc9e8ec Jul 31 01:03:16.202484 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:16.202449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" event={"ID":"9f7ee2f2-6fdc-430b-88e2-c1047a19df31","Type":"ContainerStarted","Data":"fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb"} Jul 31 01:03:16.202887 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:16.202494 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" event={"ID":"9f7ee2f2-6fdc-430b-88e2-c1047a19df31","Type":"ContainerStarted","Data":"6e0a7d1dceb986dd58ac90b5763dd82a271fbc0293975c09a13e6fa7acc9e8ec"} Jul 31 01:03:16.219853 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:16.219795 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" podStartSLOduration=1.21977835 podStartE2EDuration="1.21977835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:03:16.218341347 +0000 UTC m=+3405.741309093" watchObservedRunningTime="2026-07-31 01:03:16.21977835 +0000 UTC m=+3405.742746095" Jul 31 01:03:16.875381 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:16.875345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:03:25.403370 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:25.403275 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:25.403370 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:25.403332 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:25.415926 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:25.415894 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:26.265703 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:26.265674 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:03:54.679092 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:03:54.679053 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:04:05.786473 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:05.786432 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:04:30.520616 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:30.520575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:04:33.682136 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:33.682082 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:04:38.408851 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:38.408808 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:04:39.373225 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:39.373187 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz"] Jul 31 01:04:39.390286 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:39.390249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:04:41.521862 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:41.521827 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:04:42.366676 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.366637 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc"] Jul 31 01:04:42.370376 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.370349 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.373068 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.373044 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6a2164\"/\"kube-root-ca.crt\"" Jul 31 01:04:42.374126 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.374097 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6a2164\"/\"rate-test-model-6a2164-kserve-self-signed-certs\"" Jul 31 01:04:42.374215 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.374193 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6a2164\"/\"default-dockercfg-7chb9\"" Jul 31 01:04:42.374291 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.374200 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6a2164\"/\"openshift-service-ca.crt\"" Jul 31 01:04:42.381532 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.381508 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc"] Jul 31 01:04:42.389278 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.389247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:04:42.454852 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.454807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-model-cache\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.454852 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.454855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f0151-63cd-4922-8d7f-e23297efe3f6-tls-certs\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.455106 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.454882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzw7f\" (UniqueName: \"kubernetes.io/projected/c81f0151-63cd-4922-8d7f-e23297efe3f6-kube-api-access-xzw7f\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.455106 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.454917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-home\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.455106 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.455025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-tmp-dir\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.455106 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.455055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-dshm\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.555796 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.555760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-model-cache\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.555796 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.555795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f0151-63cd-4922-8d7f-e23297efe3f6-tls-certs\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.556342 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.555818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzw7f\" (UniqueName: \"kubernetes.io/projected/c81f0151-63cd-4922-8d7f-e23297efe3f6-kube-api-access-xzw7f\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.556342 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.555862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-home\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.556342 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.555907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-tmp-dir\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.556342 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.555933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-dshm\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.556342 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.556295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-model-cache\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.556342 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.556318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-home\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.556586 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.556372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-tmp-dir\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.558235 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.558211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-dshm\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.558417 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.558398 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f0151-63cd-4922-8d7f-e23297efe3f6-tls-certs\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.563615 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.563595 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzw7f\" (UniqueName: \"kubernetes.io/projected/c81f0151-63cd-4922-8d7f-e23297efe3f6-kube-api-access-xzw7f\") pod \"rate-test-model-6a2164-kserve-995cd68b7-46nvc\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.680917 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.680854 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:42.820850 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:42.820823 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc"] Jul 31 01:04:42.823176 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:04:42.823148 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc81f0151_63cd_4922_8d7f_e23297efe3f6.slice/crio-c37a2ed3fac542e1b9490da30c280b4aa09967c323bd1f4df79c303358f4ff02 WatchSource:0}: Error finding container c37a2ed3fac542e1b9490da30c280b4aa09967c323bd1f4df79c303358f4ff02: Status 404 returned error can't find the container with id c37a2ed3fac542e1b9490da30c280b4aa09967c323bd1f4df79c303358f4ff02 Jul 31 01:04:43.649009 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:43.648970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" event={"ID":"c81f0151-63cd-4922-8d7f-e23297efe3f6","Type":"ContainerStarted","Data":"fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07"} Jul 31 01:04:43.649499 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:43.649020 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" event={"ID":"c81f0151-63cd-4922-8d7f-e23297efe3f6","Type":"ContainerStarted","Data":"c37a2ed3fac542e1b9490da30c280b4aa09967c323bd1f4df79c303358f4ff02"} Jul 31 01:04:43.666861 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:43.666802 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" podStartSLOduration=1.666784896 podStartE2EDuration="1.666784896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:04:43.66395012 +0000 UTC m=+3493.186917866" watchObservedRunningTime="2026-07-31 01:04:43.666784896 +0000 UTC m=+3493.189752697" Jul 31 01:04:45.089314 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:45.089277 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:04:52.681089 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:52.680993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:52.681089 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:52.681072 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:52.694074 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:52.694045 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:52.707155 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:52.707106 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:04:54.393387 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.393346 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" podUID="9f7ee2f2-6fdc-430b-88e2-c1047a19df31" containerName="main" containerID="cri-o://fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb" gracePeriod=45 Jul 31 01:04:54.644390 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.644323 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:04:54.707410 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.707373 2575 generic.go:356] "Generic (PLEG): container finished" podID="9f7ee2f2-6fdc-430b-88e2-c1047a19df31" containerID="fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb" exitCode=0 Jul 31 01:04:54.707612 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.707420 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" Jul 31 01:04:54.707612 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.707451 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" event={"ID":"9f7ee2f2-6fdc-430b-88e2-c1047a19df31","Type":"ContainerDied","Data":"fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb"} Jul 31 01:04:54.707612 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.707489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz" event={"ID":"9f7ee2f2-6fdc-430b-88e2-c1047a19df31","Type":"ContainerDied","Data":"6e0a7d1dceb986dd58ac90b5763dd82a271fbc0293975c09a13e6fa7acc9e8ec"} Jul 31 01:04:54.707612 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.707507 2575 scope.go:117] "RemoveContainer" containerID="fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb" Jul 31 01:04:54.716082 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.716062 2575 scope.go:117] "RemoveContainer" containerID="fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb" Jul 31 01:04:54.716394 ip-10-0-134-191 kubenswrapper[2575]: E0731 01:04:54.716368 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb\": container with ID starting with fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb not found: ID does not exist" containerID="fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb" Jul 31 01:04:54.716448 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.716405 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb"} err="failed to get container status \"fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb\": rpc error: code = NotFound desc = could not find container \"fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb\": container with ID starting with fd328cf09660f1b2746d5c289544e1840973fe86d1579fdd068ea92ecb4315fb not found: ID does not exist" Jul 31 01:04:54.763477 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763439 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tmp-dir\") pod \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " Jul 31 01:04:54.763678 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763492 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tls-certs\") pod \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " Jul 31 01:04:54.763678 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763521 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-dshm\") pod \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " Jul 31 01:04:54.763678 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763547 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-model-cache\") pod \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " Jul 31 01:04:54.763678 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763607 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc2rn\" (UniqueName: \"kubernetes.io/projected/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-kube-api-access-fc2rn\") pod \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " Jul 31 01:04:54.763678 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763676 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-home\") pod \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\" (UID: \"9f7ee2f2-6fdc-430b-88e2-c1047a19df31\") " Jul 31 01:04:54.763930 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763719 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f7ee2f2-6fdc-430b-88e2-c1047a19df31" (UID: "9f7ee2f2-6fdc-430b-88e2-c1047a19df31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:04:54.763930 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763824 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f7ee2f2-6fdc-430b-88e2-c1047a19df31" (UID: "9f7ee2f2-6fdc-430b-88e2-c1047a19df31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:04:54.764025 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.763957 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-home" (OuterVolumeSpecName: "home") pod "9f7ee2f2-6fdc-430b-88e2-c1047a19df31" (UID: "9f7ee2f2-6fdc-430b-88e2-c1047a19df31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:04:54.764081 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.764036 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tmp-dir\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:04:54.764081 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.764058 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-model-cache\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:04:54.765715 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.765686 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-dshm" (OuterVolumeSpecName: "dshm") pod "9f7ee2f2-6fdc-430b-88e2-c1047a19df31" (UID: "9f7ee2f2-6fdc-430b-88e2-c1047a19df31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:04:54.766133 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.766092 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f7ee2f2-6fdc-430b-88e2-c1047a19df31" (UID: "9f7ee2f2-6fdc-430b-88e2-c1047a19df31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 01:04:54.766229 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.766147 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-kube-api-access-fc2rn" (OuterVolumeSpecName: "kube-api-access-fc2rn") pod "9f7ee2f2-6fdc-430b-88e2-c1047a19df31" (UID: "9f7ee2f2-6fdc-430b-88e2-c1047a19df31"). InnerVolumeSpecName "kube-api-access-fc2rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:04:54.865455 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.865421 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-home\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:04:54.865455 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.865453 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-tls-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:04:54.865455 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.865464 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-dshm\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:04:54.865699 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:54.865473 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc2rn\" (UniqueName: \"kubernetes.io/projected/9f7ee2f2-6fdc-430b-88e2-c1047a19df31-kube-api-access-fc2rn\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:04:55.029372 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:55.029301 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz"] Jul 31 01:04:55.031225 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:55.031199 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b32567/sub-test-model-b32567-kserve-6b4d98bfd5-hmfgz"] Jul 31 01:04:55.049406 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:04:55.049374 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7ee2f2-6fdc-430b-88e2-c1047a19df31" path="/var/lib/kubelet/pods/9f7ee2f2-6fdc-430b-88e2-c1047a19df31/volumes" Jul 31 01:05:01.284751 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:05:01.284672 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:05:03.978981 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:05:03.978940 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:05:46.775936 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:05:46.775889 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:05:55.378270 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:05:55.378226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:06:10.877329 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:10.877293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:06:16.462935 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:16.462901 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc"] Jul 31 01:06:16.483578 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:16.483549 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:17.480581 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:17.480546 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:18.532127 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.532075 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7581-140389896645808"] Jul 31 01:06:18.532545 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.532487 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7ee2f2-6fdc-430b-88e2-c1047a19df31" containerName="main" Jul 31 01:06:18.532545 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.532499 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7ee2f2-6fdc-430b-88e2-c1047a19df31" containerName="main" Jul 31 01:06:18.532619 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.532556 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7ee2f2-6fdc-430b-88e2-c1047a19df31" containerName="main" Jul 31 01:06:18.535584 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.535565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896645808" Jul 31 01:06:18.538099 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.538076 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7trdk\"" Jul 31 01:06:18.540011 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.539986 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896645808"] Jul 31 01:06:18.667501 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.667454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntjzk\" (UniqueName: \"kubernetes.io/projected/46a20ac5-f97e-4bc3-9c20-64e941b09199-kube-api-access-ntjzk\") pod \"test-curl-7581-140389896645808\" (UID: \"46a20ac5-f97e-4bc3-9c20-64e941b09199\") " pod="opendatahub/test-curl-7581-140389896645808" Jul 31 01:06:18.768504 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.768467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjzk\" (UniqueName: \"kubernetes.io/projected/46a20ac5-f97e-4bc3-9c20-64e941b09199-kube-api-access-ntjzk\") pod \"test-curl-7581-140389896645808\" (UID: \"46a20ac5-f97e-4bc3-9c20-64e941b09199\") " pod="opendatahub/test-curl-7581-140389896645808" Jul 31 01:06:18.776939 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.776905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjzk\" (UniqueName: \"kubernetes.io/projected/46a20ac5-f97e-4bc3-9c20-64e941b09199-kube-api-access-ntjzk\") pod \"test-curl-7581-140389896645808\" (UID: \"46a20ac5-f97e-4bc3-9c20-64e941b09199\") " pod="opendatahub/test-curl-7581-140389896645808" Jul 31 01:06:18.847429 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.847328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896645808" Jul 31 01:06:18.988997 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.988975 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896645808"] Jul 31 01:06:18.990851 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:06:18.990826 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46a20ac5_f97e_4bc3_9c20_64e941b09199.slice/crio-a1e31b4829890920e35a8ee638b0675443796660edf7a8fbf5123d19291e6945 WatchSource:0}: Error finding container a1e31b4829890920e35a8ee638b0675443796660edf7a8fbf5123d19291e6945: Status 404 returned error can't find the container with id a1e31b4829890920e35a8ee638b0675443796660edf7a8fbf5123d19291e6945 Jul 31 01:06:18.992577 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:18.992555 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 01:06:19.137817 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:19.137729 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896645808" event={"ID":"46a20ac5-f97e-4bc3-9c20-64e941b09199","Type":"ContainerStarted","Data":"a1e31b4829890920e35a8ee638b0675443796660edf7a8fbf5123d19291e6945"} Jul 31 01:06:20.143928 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:20.143838 2575 generic.go:356] "Generic (PLEG): container finished" podID="46a20ac5-f97e-4bc3-9c20-64e941b09199" containerID="d83f468a0b8ec1386563f1e93825d06e0a910c7b86c50a0316ebbc35fefd0ba7" exitCode=0 Jul 31 01:06:20.144323 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:20.143931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896645808" event={"ID":"46a20ac5-f97e-4bc3-9c20-64e941b09199","Type":"ContainerDied","Data":"d83f468a0b8ec1386563f1e93825d06e0a910c7b86c50a0316ebbc35fefd0ba7"} Jul 31 01:06:20.577225 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:20.577189 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:06:21.296323 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.296289 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896645808" Jul 31 01:06:21.314823 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.314788 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7581-140389896645808_46a20ac5-f97e-4bc3-9c20-64e941b09199/test-curl-7581-140389896645808/0.log" Jul 31 01:06:21.337659 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.337623 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7581-140389896645808"] Jul 31 01:06:21.343815 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.343780 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7581-140389896645808"] Jul 31 01:06:21.394244 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.394208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntjzk\" (UniqueName: \"kubernetes.io/projected/46a20ac5-f97e-4bc3-9c20-64e941b09199-kube-api-access-ntjzk\") pod \"46a20ac5-f97e-4bc3-9c20-64e941b09199\" (UID: \"46a20ac5-f97e-4bc3-9c20-64e941b09199\") " Jul 31 01:06:21.396450 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.396426 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a20ac5-f97e-4bc3-9c20-64e941b09199-kube-api-access-ntjzk" (OuterVolumeSpecName: "kube-api-access-ntjzk") pod "46a20ac5-f97e-4bc3-9c20-64e941b09199" (UID: "46a20ac5-f97e-4bc3-9c20-64e941b09199"). InnerVolumeSpecName "kube-api-access-ntjzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:06:21.467668 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.467631 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7581-140389896644400"] Jul 31 01:06:21.468011 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.467997 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a20ac5-f97e-4bc3-9c20-64e941b09199" containerName="test-curl-7581-140389896645808" Jul 31 01:06:21.468060 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.468012 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a20ac5-f97e-4bc3-9c20-64e941b09199" containerName="test-curl-7581-140389896645808" Jul 31 01:06:21.468099 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.468091 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46a20ac5-f97e-4bc3-9c20-64e941b09199" containerName="test-curl-7581-140389896645808" Jul 31 01:06:21.471136 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.471085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896644400" Jul 31 01:06:21.477001 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.476969 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896644400"] Jul 31 01:06:21.495074 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.495041 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntjzk\" (UniqueName: \"kubernetes.io/projected/46a20ac5-f97e-4bc3-9c20-64e941b09199-kube-api-access-ntjzk\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:21.595675 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.595639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szt7p\" (UniqueName: \"kubernetes.io/projected/82b04687-cb83-48ca-93e6-fdff64c1eca1-kube-api-access-szt7p\") pod \"test-curl-7581-140389896644400\" (UID: \"82b04687-cb83-48ca-93e6-fdff64c1eca1\") " pod="opendatahub/test-curl-7581-140389896644400" Jul 31 01:06:21.696527 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.696429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szt7p\" (UniqueName: \"kubernetes.io/projected/82b04687-cb83-48ca-93e6-fdff64c1eca1-kube-api-access-szt7p\") pod \"test-curl-7581-140389896644400\" (UID: \"82b04687-cb83-48ca-93e6-fdff64c1eca1\") " pod="opendatahub/test-curl-7581-140389896644400" Jul 31 01:06:21.704113 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.704082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szt7p\" (UniqueName: \"kubernetes.io/projected/82b04687-cb83-48ca-93e6-fdff64c1eca1-kube-api-access-szt7p\") pod \"test-curl-7581-140389896644400\" (UID: \"82b04687-cb83-48ca-93e6-fdff64c1eca1\") " pod="opendatahub/test-curl-7581-140389896644400" Jul 31 01:06:21.782004 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.781968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896644400" Jul 31 01:06:21.922804 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:21.922773 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896644400"] Jul 31 01:06:21.924245 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:06:21.924215 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82b04687_cb83_48ca_93e6_fdff64c1eca1.slice/crio-fe246e72f987224a8a4e4a579ccfa2316b7e604e3e1c425d4aa2f73823c1fd39 WatchSource:0}: Error finding container fe246e72f987224a8a4e4a579ccfa2316b7e604e3e1c425d4aa2f73823c1fd39: Status 404 returned error can't find the container with id fe246e72f987224a8a4e4a579ccfa2316b7e604e3e1c425d4aa2f73823c1fd39 Jul 31 01:06:22.163556 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:22.163520 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896645808" Jul 31 01:06:22.163556 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:22.163547 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e31b4829890920e35a8ee638b0675443796660edf7a8fbf5123d19291e6945" Jul 31 01:06:22.164879 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:22.164846 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896644400" event={"ID":"82b04687-cb83-48ca-93e6-fdff64c1eca1","Type":"ContainerStarted","Data":"fe246e72f987224a8a4e4a579ccfa2316b7e604e3e1c425d4aa2f73823c1fd39"} Jul 31 01:06:22.174920 ip-10-0-134-191 kubenswrapper[2575]: E0731 01:06:22.174888 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7581-140389896645808\" is forbidden: User \"system:node:ip-10-0-134-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-191.ec2.internal' and this object" podUID="46a20ac5-f97e-4bc3-9c20-64e941b09199" pod="opendatahub/test-curl-7581-140389896645808" Jul 31 01:06:23.049029 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:23.048996 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46a20ac5-f97e-4bc3-9c20-64e941b09199" path="/var/lib/kubelet/pods/46a20ac5-f97e-4bc3-9c20-64e941b09199/volumes" Jul 31 01:06:23.170835 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:23.170801 2575 generic.go:356] "Generic (PLEG): container finished" podID="82b04687-cb83-48ca-93e6-fdff64c1eca1" containerID="1ac57d0841c4b752cdbacc108a01e1a9843ecee474be3bfa6a16c4c16ea275d1" exitCode=0 Jul 31 01:06:23.171006 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:23.170890 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896644400" event={"ID":"82b04687-cb83-48ca-93e6-fdff64c1eca1","Type":"ContainerDied","Data":"1ac57d0841c4b752cdbacc108a01e1a9843ecee474be3bfa6a16c4c16ea275d1"} Jul 31 01:06:24.315408 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.315379 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896644400" Jul 31 01:06:24.334575 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.334543 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7581-140389896644400_82b04687-cb83-48ca-93e6-fdff64c1eca1/test-curl-7581-140389896644400/0.log" Jul 31 01:06:24.357538 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.357491 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7581-140389896644400"] Jul 31 01:06:24.359514 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.359489 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7581-140389896644400"] Jul 31 01:06:24.422822 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.422785 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szt7p\" (UniqueName: \"kubernetes.io/projected/82b04687-cb83-48ca-93e6-fdff64c1eca1-kube-api-access-szt7p\") pod \"82b04687-cb83-48ca-93e6-fdff64c1eca1\" (UID: \"82b04687-cb83-48ca-93e6-fdff64c1eca1\") " Jul 31 01:06:24.424977 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.424946 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b04687-cb83-48ca-93e6-fdff64c1eca1-kube-api-access-szt7p" (OuterVolumeSpecName: "kube-api-access-szt7p") pod "82b04687-cb83-48ca-93e6-fdff64c1eca1" (UID: "82b04687-cb83-48ca-93e6-fdff64c1eca1"). InnerVolumeSpecName "kube-api-access-szt7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:06:24.524031 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.523992 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szt7p\" (UniqueName: \"kubernetes.io/projected/82b04687-cb83-48ca-93e6-fdff64c1eca1-kube-api-access-szt7p\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:24.530239 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.530206 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:24.530596 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.530582 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b04687-cb83-48ca-93e6-fdff64c1eca1" containerName="test-curl-7581-140389896644400" Jul 31 01:06:24.530642 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.530597 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b04687-cb83-48ca-93e6-fdff64c1eca1" containerName="test-curl-7581-140389896644400" Jul 31 01:06:24.530678 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.530662 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b04687-cb83-48ca-93e6-fdff64c1eca1" containerName="test-curl-7581-140389896644400" Jul 31 01:06:24.533694 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.533673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:24.539713 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.539687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:24.624710 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.624663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfznd\" (UniqueName: \"kubernetes.io/projected/007dd0fe-a741-4267-8dd3-cdf226ed63d8-kube-api-access-qfznd\") pod \"test-curl-7581-140389896813712\" (UID: \"007dd0fe-a741-4267-8dd3-cdf226ed63d8\") " pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:24.725547 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.725447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfznd\" (UniqueName: \"kubernetes.io/projected/007dd0fe-a741-4267-8dd3-cdf226ed63d8-kube-api-access-qfznd\") pod \"test-curl-7581-140389896813712\" (UID: \"007dd0fe-a741-4267-8dd3-cdf226ed63d8\") " pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:24.734444 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.734409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfznd\" (UniqueName: \"kubernetes.io/projected/007dd0fe-a741-4267-8dd3-cdf226ed63d8-kube-api-access-qfznd\") pod \"test-curl-7581-140389896813712\" (UID: \"007dd0fe-a741-4267-8dd3-cdf226ed63d8\") " pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:24.845642 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.845603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:24.983563 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:24.983533 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:24.985360 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:06:24.985327 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod007dd0fe_a741_4267_8dd3_cdf226ed63d8.slice/crio-77524ab014b608282c11d8a7fb954f3311f2ae431ce60f16974eb30671087c9e WatchSource:0}: Error finding container 77524ab014b608282c11d8a7fb954f3311f2ae431ce60f16974eb30671087c9e: Status 404 returned error can't find the container with id 77524ab014b608282c11d8a7fb954f3311f2ae431ce60f16974eb30671087c9e Jul 31 01:06:25.049416 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:25.049379 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b04687-cb83-48ca-93e6-fdff64c1eca1" path="/var/lib/kubelet/pods/82b04687-cb83-48ca-93e6-fdff64c1eca1/volumes" Jul 31 01:06:25.183013 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:25.182985 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896644400" Jul 31 01:06:25.183224 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:25.182997 2575 scope.go:117] "RemoveContainer" containerID="1ac57d0841c4b752cdbacc108a01e1a9843ecee474be3bfa6a16c4c16ea275d1" Jul 31 01:06:25.184401 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:25.184224 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896813712" event={"ID":"007dd0fe-a741-4267-8dd3-cdf226ed63d8","Type":"ContainerStarted","Data":"77524ab014b608282c11d8a7fb954f3311f2ae431ce60f16974eb30671087c9e"} Jul 31 01:06:26.190974 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:26.190938 2575 generic.go:356] "Generic (PLEG): container finished" podID="007dd0fe-a741-4267-8dd3-cdf226ed63d8" containerID="7728e34a7417af6fe56cee29f02fc5c0910807ff9f9868522f9d5bf5df204a96" exitCode=0 Jul 31 01:06:26.191469 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:26.191023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896813712" event={"ID":"007dd0fe-a741-4267-8dd3-cdf226ed63d8","Type":"ContainerDied","Data":"7728e34a7417af6fe56cee29f02fc5c0910807ff9f9868522f9d5bf5df204a96"} Jul 31 01:06:27.337334 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.337306 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:27.361379 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.361346 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7581-140389896813712_007dd0fe-a741-4267-8dd3-cdf226ed63d8/test-curl-7581-140389896813712/0.log" Jul 31 01:06:27.384260 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.384223 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:27.386145 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.386100 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:27.448877 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.448788 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfznd\" (UniqueName: \"kubernetes.io/projected/007dd0fe-a741-4267-8dd3-cdf226ed63d8-kube-api-access-qfznd\") pod \"007dd0fe-a741-4267-8dd3-cdf226ed63d8\" (UID: \"007dd0fe-a741-4267-8dd3-cdf226ed63d8\") " Jul 31 01:06:27.450857 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.450826 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007dd0fe-a741-4267-8dd3-cdf226ed63d8-kube-api-access-qfznd" (OuterVolumeSpecName: "kube-api-access-qfznd") pod "007dd0fe-a741-4267-8dd3-cdf226ed63d8" (UID: "007dd0fe-a741-4267-8dd3-cdf226ed63d8"). InnerVolumeSpecName "kube-api-access-qfznd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:06:27.505471 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.505436 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:27.505841 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.505825 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007dd0fe-a741-4267-8dd3-cdf226ed63d8" containerName="test-curl-7581-140389896813712" Jul 31 01:06:27.505902 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.505842 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="007dd0fe-a741-4267-8dd3-cdf226ed63d8" containerName="test-curl-7581-140389896813712" Jul 31 01:06:27.505960 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.505921 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="007dd0fe-a741-4267-8dd3-cdf226ed63d8" containerName="test-curl-7581-140389896813712" Jul 31 01:06:27.508996 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.508967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:27.514926 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.514898 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:27.550175 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.550135 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfznd\" (UniqueName: \"kubernetes.io/projected/007dd0fe-a741-4267-8dd3-cdf226ed63d8-kube-api-access-qfznd\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:27.651113 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.651069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jkv\" (UniqueName: \"kubernetes.io/projected/5f8a4202-ca59-4555-a0d6-86a7ed4bcc86-kube-api-access-q9jkv\") pod \"test-curl-7581-140389896813712\" (UID: \"5f8a4202-ca59-4555-a0d6-86a7ed4bcc86\") " pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:27.751997 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.751907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jkv\" (UniqueName: \"kubernetes.io/projected/5f8a4202-ca59-4555-a0d6-86a7ed4bcc86-kube-api-access-q9jkv\") pod \"test-curl-7581-140389896813712\" (UID: \"5f8a4202-ca59-4555-a0d6-86a7ed4bcc86\") " pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:27.759620 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.759591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jkv\" (UniqueName: \"kubernetes.io/projected/5f8a4202-ca59-4555-a0d6-86a7ed4bcc86-kube-api-access-q9jkv\") pod \"test-curl-7581-140389896813712\" (UID: \"5f8a4202-ca59-4555-a0d6-86a7ed4bcc86\") " pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:27.819909 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.819863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:27.957977 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:27.957950 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:27.960215 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:06:27.960187 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f8a4202_ca59_4555_a0d6_86a7ed4bcc86.slice/crio-1ca7835605b348bbe745ff17b00bd3d0f8e11c0b9dfb5f0af4a1da2376ae09a0 WatchSource:0}: Error finding container 1ca7835605b348bbe745ff17b00bd3d0f8e11c0b9dfb5f0af4a1da2376ae09a0: Status 404 returned error can't find the container with id 1ca7835605b348bbe745ff17b00bd3d0f8e11c0b9dfb5f0af4a1da2376ae09a0 Jul 31 01:06:28.204687 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:28.204648 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896813712" event={"ID":"5f8a4202-ca59-4555-a0d6-86a7ed4bcc86","Type":"ContainerStarted","Data":"1ca7835605b348bbe745ff17b00bd3d0f8e11c0b9dfb5f0af4a1da2376ae09a0"} Jul 31 01:06:28.205777 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:28.205753 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77524ab014b608282c11d8a7fb954f3311f2ae431ce60f16974eb30671087c9e" Jul 31 01:06:28.205904 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:28.205787 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:28.217453 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:28.217418 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7581-140389896813712" oldPodUID="007dd0fe-a741-4267-8dd3-cdf226ed63d8" podUID="5f8a4202-ca59-4555-a0d6-86a7ed4bcc86" Jul 31 01:06:29.049809 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:29.049774 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="007dd0fe-a741-4267-8dd3-cdf226ed63d8" path="/var/lib/kubelet/pods/007dd0fe-a741-4267-8dd3-cdf226ed63d8/volumes" Jul 31 01:06:29.212684 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:29.212651 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f8a4202-ca59-4555-a0d6-86a7ed4bcc86" containerID="0f2877af9b87629854afc5f541c13ab468e428981a0808ca1fe0c495c1defe21" exitCode=0 Jul 31 01:06:29.212868 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:29.212743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896813712" event={"ID":"5f8a4202-ca59-4555-a0d6-86a7ed4bcc86","Type":"ContainerDied","Data":"0f2877af9b87629854afc5f541c13ab468e428981a0808ca1fe0c495c1defe21"} Jul 31 01:06:30.357175 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.357148 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:30.375787 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.375758 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7581-140389896813712_5f8a4202-ca59-4555-a0d6-86a7ed4bcc86/test-curl-7581-140389896813712/0.log" Jul 31 01:06:30.396551 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.396516 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:30.399587 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.399559 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7581-140389896813712"] Jul 31 01:06:30.476108 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.476016 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9jkv\" (UniqueName: \"kubernetes.io/projected/5f8a4202-ca59-4555-a0d6-86a7ed4bcc86-kube-api-access-q9jkv\") pod \"5f8a4202-ca59-4555-a0d6-86a7ed4bcc86\" (UID: \"5f8a4202-ca59-4555-a0d6-86a7ed4bcc86\") " Jul 31 01:06:30.478163 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.478099 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8a4202-ca59-4555-a0d6-86a7ed4bcc86-kube-api-access-q9jkv" (OuterVolumeSpecName: "kube-api-access-q9jkv") pod "5f8a4202-ca59-4555-a0d6-86a7ed4bcc86" (UID: "5f8a4202-ca59-4555-a0d6-86a7ed4bcc86"). InnerVolumeSpecName "kube-api-access-q9jkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:06:30.516961 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.516921 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:30.517368 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.517346 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8a4202-ca59-4555-a0d6-86a7ed4bcc86" containerName="test-curl-7581-140389896813712" Jul 31 01:06:30.517449 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.517371 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8a4202-ca59-4555-a0d6-86a7ed4bcc86" containerName="test-curl-7581-140389896813712" Jul 31 01:06:30.517506 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.517495 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8a4202-ca59-4555-a0d6-86a7ed4bcc86" containerName="test-curl-7581-140389896813712" Jul 31 01:06:30.520695 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.520671 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:30.524881 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.524854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:30.577685 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.577650 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9jkv\" (UniqueName: \"kubernetes.io/projected/5f8a4202-ca59-4555-a0d6-86a7ed4bcc86-kube-api-access-q9jkv\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:30.679094 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.679048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcpnr\" (UniqueName: \"kubernetes.io/projected/fc0af5ef-548f-4451-a25f-ec4fdb63a04d-kube-api-access-zcpnr\") pod \"test-curl-7581-140389896667184\" (UID: \"fc0af5ef-548f-4451-a25f-ec4fdb63a04d\") " pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:30.779661 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.779569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcpnr\" (UniqueName: \"kubernetes.io/projected/fc0af5ef-548f-4451-a25f-ec4fdb63a04d-kube-api-access-zcpnr\") pod \"test-curl-7581-140389896667184\" (UID: \"fc0af5ef-548f-4451-a25f-ec4fdb63a04d\") " pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:30.788471 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.788443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcpnr\" (UniqueName: \"kubernetes.io/projected/fc0af5ef-548f-4451-a25f-ec4fdb63a04d-kube-api-access-zcpnr\") pod \"test-curl-7581-140389896667184\" (UID: \"fc0af5ef-548f-4451-a25f-ec4fdb63a04d\") " pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:30.830630 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.830600 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:30.974247 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:30.974214 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:30.976249 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:06:30.976225 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc0af5ef_548f_4451_a25f_ec4fdb63a04d.slice/crio-222168e48fe07ae145428df6de9155a6a8b5dbd795420eff7e007a40aca03079 WatchSource:0}: Error finding container 222168e48fe07ae145428df6de9155a6a8b5dbd795420eff7e007a40aca03079: Status 404 returned error can't find the container with id 222168e48fe07ae145428df6de9155a6a8b5dbd795420eff7e007a40aca03079 Jul 31 01:06:31.055968 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.055887 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f8a4202-ca59-4555-a0d6-86a7ed4bcc86" path="/var/lib/kubelet/pods/5f8a4202-ca59-4555-a0d6-86a7ed4bcc86/volumes" Jul 31 01:06:31.223171 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.223143 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896813712" Jul 31 01:06:31.223362 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.223152 2575 scope.go:117] "RemoveContainer" containerID="0f2877af9b87629854afc5f541c13ab468e428981a0808ca1fe0c495c1defe21" Jul 31 01:06:31.224472 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.224371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896667184" event={"ID":"fc0af5ef-548f-4451-a25f-ec4fdb63a04d","Type":"ContainerStarted","Data":"222168e48fe07ae145428df6de9155a6a8b5dbd795420eff7e007a40aca03079"} Jul 31 01:06:31.482922 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.482879 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" podUID="c81f0151-63cd-4922-8d7f-e23297efe3f6" containerName="main" containerID="cri-o://fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07" gracePeriod=45 Jul 31 01:06:31.745353 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.745328 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:06:31.889788 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.889695 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzw7f\" (UniqueName: \"kubernetes.io/projected/c81f0151-63cd-4922-8d7f-e23297efe3f6-kube-api-access-xzw7f\") pod \"c81f0151-63cd-4922-8d7f-e23297efe3f6\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " Jul 31 01:06:31.889788 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.889740 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-dshm\") pod \"c81f0151-63cd-4922-8d7f-e23297efe3f6\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " Jul 31 01:06:31.889998 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.889818 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-tmp-dir\") pod \"c81f0151-63cd-4922-8d7f-e23297efe3f6\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " Jul 31 01:06:31.889998 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.889863 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-model-cache\") pod \"c81f0151-63cd-4922-8d7f-e23297efe3f6\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " Jul 31 01:06:31.889998 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.889927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-home\") pod \"c81f0151-63cd-4922-8d7f-e23297efe3f6\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " Jul 31 01:06:31.889998 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.889955 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f0151-63cd-4922-8d7f-e23297efe3f6-tls-certs\") pod \"c81f0151-63cd-4922-8d7f-e23297efe3f6\" (UID: \"c81f0151-63cd-4922-8d7f-e23297efe3f6\") " Jul 31 01:06:31.890243 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.890105 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c81f0151-63cd-4922-8d7f-e23297efe3f6" (UID: "c81f0151-63cd-4922-8d7f-e23297efe3f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:06:31.890243 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.890166 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c81f0151-63cd-4922-8d7f-e23297efe3f6" (UID: "c81f0151-63cd-4922-8d7f-e23297efe3f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:06:31.890350 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.890269 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-home" (OuterVolumeSpecName: "home") pod "c81f0151-63cd-4922-8d7f-e23297efe3f6" (UID: "c81f0151-63cd-4922-8d7f-e23297efe3f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:06:31.890442 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.890425 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-model-cache\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:31.890497 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.890448 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-home\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:31.890497 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.890463 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-tmp-dir\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:31.891962 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.891938 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-dshm" (OuterVolumeSpecName: "dshm") pod "c81f0151-63cd-4922-8d7f-e23297efe3f6" (UID: "c81f0151-63cd-4922-8d7f-e23297efe3f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:06:31.892405 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.892385 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81f0151-63cd-4922-8d7f-e23297efe3f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c81f0151-63cd-4922-8d7f-e23297efe3f6" (UID: "c81f0151-63cd-4922-8d7f-e23297efe3f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 01:06:31.892472 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.892405 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81f0151-63cd-4922-8d7f-e23297efe3f6-kube-api-access-xzw7f" (OuterVolumeSpecName: "kube-api-access-xzw7f") pod "c81f0151-63cd-4922-8d7f-e23297efe3f6" (UID: "c81f0151-63cd-4922-8d7f-e23297efe3f6"). InnerVolumeSpecName "kube-api-access-xzw7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:06:31.991496 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.991434 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81f0151-63cd-4922-8d7f-e23297efe3f6-tls-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:31.991496 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.991485 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzw7f\" (UniqueName: \"kubernetes.io/projected/c81f0151-63cd-4922-8d7f-e23297efe3f6-kube-api-access-xzw7f\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:31.991496 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:31.991502 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81f0151-63cd-4922-8d7f-e23297efe3f6-dshm\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:32.230756 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.230720 2575 generic.go:356] "Generic (PLEG): container finished" podID="fc0af5ef-548f-4451-a25f-ec4fdb63a04d" containerID="1c492160a7daf9d5a6ee79ce0001e943ce58bd3d963fbb814f261d7f837bd4c6" exitCode=0 Jul 31 01:06:32.230936 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.230811 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896667184" event={"ID":"fc0af5ef-548f-4451-a25f-ec4fdb63a04d","Type":"ContainerDied","Data":"1c492160a7daf9d5a6ee79ce0001e943ce58bd3d963fbb814f261d7f837bd4c6"} Jul 31 01:06:32.232057 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.232028 2575 generic.go:356] "Generic (PLEG): container finished" podID="c81f0151-63cd-4922-8d7f-e23297efe3f6" containerID="fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07" exitCode=0 Jul 31 01:06:32.232207 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.232073 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" Jul 31 01:06:32.232207 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.232106 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" event={"ID":"c81f0151-63cd-4922-8d7f-e23297efe3f6","Type":"ContainerDied","Data":"fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07"} Jul 31 01:06:32.232207 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.232153 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc" event={"ID":"c81f0151-63cd-4922-8d7f-e23297efe3f6","Type":"ContainerDied","Data":"c37a2ed3fac542e1b9490da30c280b4aa09967c323bd1f4df79c303358f4ff02"} Jul 31 01:06:32.232207 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.232172 2575 scope.go:117] "RemoveContainer" containerID="fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07" Jul 31 01:06:32.241036 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.241009 2575 scope.go:117] "RemoveContainer" containerID="fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07" Jul 31 01:06:32.241381 ip-10-0-134-191 kubenswrapper[2575]: E0731 01:06:32.241360 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07\": container with ID starting with fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07 not found: ID does not exist" containerID="fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07" Jul 31 01:06:32.241466 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.241399 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07"} err="failed to get container status \"fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07\": rpc error: code = NotFound desc = could not find container \"fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07\": container with ID starting with fc087b649f27c6c4896be30e409b3d5f95f84ff24f481aa5641200d625798b07 not found: ID does not exist" Jul 31 01:06:32.257398 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.257365 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc"] Jul 31 01:06:32.263312 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:32.263276 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6a2164/rate-test-model-6a2164-kserve-995cd68b7-46nvc"] Jul 31 01:06:33.049265 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.049230 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c81f0151-63cd-4922-8d7f-e23297efe3f6" path="/var/lib/kubelet/pods/c81f0151-63cd-4922-8d7f-e23297efe3f6/volumes" Jul 31 01:06:33.375999 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.375972 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:33.395993 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.395947 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7581-140389896667184_fc0af5ef-548f-4451-a25f-ec4fdb63a04d/test-curl-7581-140389896667184/0.log" Jul 31 01:06:33.420626 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.420589 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:33.423771 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.423741 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:33.505022 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.504986 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcpnr\" (UniqueName: \"kubernetes.io/projected/fc0af5ef-548f-4451-a25f-ec4fdb63a04d-kube-api-access-zcpnr\") pod \"fc0af5ef-548f-4451-a25f-ec4fdb63a04d\" (UID: \"fc0af5ef-548f-4451-a25f-ec4fdb63a04d\") " Jul 31 01:06:33.507147 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.507096 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0af5ef-548f-4451-a25f-ec4fdb63a04d-kube-api-access-zcpnr" (OuterVolumeSpecName: "kube-api-access-zcpnr") pod "fc0af5ef-548f-4451-a25f-ec4fdb63a04d" (UID: "fc0af5ef-548f-4451-a25f-ec4fdb63a04d"). InnerVolumeSpecName "kube-api-access-zcpnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:06:33.542085 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.542047 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:33.542431 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.542417 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0af5ef-548f-4451-a25f-ec4fdb63a04d" containerName="test-curl-7581-140389896667184" Jul 31 01:06:33.542471 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.542432 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0af5ef-548f-4451-a25f-ec4fdb63a04d" containerName="test-curl-7581-140389896667184" Jul 31 01:06:33.542471 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.542449 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c81f0151-63cd-4922-8d7f-e23297efe3f6" containerName="main" Jul 31 01:06:33.542471 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.542455 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c81f0151-63cd-4922-8d7f-e23297efe3f6" containerName="main" Jul 31 01:06:33.542589 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.542515 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c81f0151-63cd-4922-8d7f-e23297efe3f6" containerName="main" Jul 31 01:06:33.542589 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.542524 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0af5ef-548f-4451-a25f-ec4fdb63a04d" containerName="test-curl-7581-140389896667184" Jul 31 01:06:33.545337 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.545317 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:33.548986 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.548954 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:33.606579 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.606489 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcpnr\" (UniqueName: \"kubernetes.io/projected/fc0af5ef-548f-4451-a25f-ec4fdb63a04d-kube-api-access-zcpnr\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:33.707817 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.707775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncrr\" (UniqueName: \"kubernetes.io/projected/c0d12e3b-b948-4524-88a2-4d79ae5c8312-kube-api-access-4ncrr\") pod \"test-curl-7581-140389896667184\" (UID: \"c0d12e3b-b948-4524-88a2-4d79ae5c8312\") " pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:33.808843 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.808801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncrr\" (UniqueName: \"kubernetes.io/projected/c0d12e3b-b948-4524-88a2-4d79ae5c8312-kube-api-access-4ncrr\") pod \"test-curl-7581-140389896667184\" (UID: \"c0d12e3b-b948-4524-88a2-4d79ae5c8312\") " pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:33.816679 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.816644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncrr\" (UniqueName: \"kubernetes.io/projected/c0d12e3b-b948-4524-88a2-4d79ae5c8312-kube-api-access-4ncrr\") pod \"test-curl-7581-140389896667184\" (UID: \"c0d12e3b-b948-4524-88a2-4d79ae5c8312\") " pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:33.856996 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.856888 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:33.994787 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:33.994757 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:33.996874 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:06:33.996839 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0d12e3b_b948_4524_88a2_4d79ae5c8312.slice/crio-8604129bf17a63edf2223a7825564aa6339c6c60f123cdd73bf5118be3a8a301 WatchSource:0}: Error finding container 8604129bf17a63edf2223a7825564aa6339c6c60f123cdd73bf5118be3a8a301: Status 404 returned error can't find the container with id 8604129bf17a63edf2223a7825564aa6339c6c60f123cdd73bf5118be3a8a301 Jul 31 01:06:34.245285 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:34.245254 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:34.245742 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:34.245258 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="222168e48fe07ae145428df6de9155a6a8b5dbd795420eff7e007a40aca03079" Jul 31 01:06:34.246447 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:34.246415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896667184" event={"ID":"c0d12e3b-b948-4524-88a2-4d79ae5c8312","Type":"ContainerStarted","Data":"8604129bf17a63edf2223a7825564aa6339c6c60f123cdd73bf5118be3a8a301"} Jul 31 01:06:34.257132 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:34.257094 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7581-140389896667184" oldPodUID="fc0af5ef-548f-4451-a25f-ec4fdb63a04d" podUID="c0d12e3b-b948-4524-88a2-4d79ae5c8312" Jul 31 01:06:35.049654 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:35.049621 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0af5ef-548f-4451-a25f-ec4fdb63a04d" path="/var/lib/kubelet/pods/fc0af5ef-548f-4451-a25f-ec4fdb63a04d/volumes" Jul 31 01:06:35.253756 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:35.253718 2575 generic.go:356] "Generic (PLEG): container finished" podID="c0d12e3b-b948-4524-88a2-4d79ae5c8312" containerID="85e1e115abbcc13161c89b59b0f25f6f100bde565ba09bb7d79978af9856b4ca" exitCode=0 Jul 31 01:06:35.254178 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:35.253808 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7581-140389896667184" event={"ID":"c0d12e3b-b948-4524-88a2-4d79ae5c8312","Type":"ContainerDied","Data":"85e1e115abbcc13161c89b59b0f25f6f100bde565ba09bb7d79978af9856b4ca"} Jul 31 01:06:36.400021 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:36.399990 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:36.421606 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:36.421573 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7581-140389896667184_c0d12e3b-b948-4524-88a2-4d79ae5c8312/test-curl-7581-140389896667184/0.log" Jul 31 01:06:36.443311 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:36.443280 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:36.445006 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:36.444983 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7581-140389896667184"] Jul 31 01:06:36.536107 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:36.536014 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ncrr\" (UniqueName: \"kubernetes.io/projected/c0d12e3b-b948-4524-88a2-4d79ae5c8312-kube-api-access-4ncrr\") pod \"c0d12e3b-b948-4524-88a2-4d79ae5c8312\" (UID: \"c0d12e3b-b948-4524-88a2-4d79ae5c8312\") " Jul 31 01:06:36.538264 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:36.538234 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d12e3b-b948-4524-88a2-4d79ae5c8312-kube-api-access-4ncrr" (OuterVolumeSpecName: "kube-api-access-4ncrr") pod "c0d12e3b-b948-4524-88a2-4d79ae5c8312" (UID: "c0d12e3b-b948-4524-88a2-4d79ae5c8312"). InnerVolumeSpecName "kube-api-access-4ncrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:06:36.636849 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:36.636808 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ncrr\" (UniqueName: \"kubernetes.io/projected/c0d12e3b-b948-4524-88a2-4d79ae5c8312-kube-api-access-4ncrr\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:06:36.940599 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:36.940562 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:37.049020 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:37.048985 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d12e3b-b948-4524-88a2-4d79ae5c8312" path="/var/lib/kubelet/pods/c0d12e3b-b948-4524-88a2-4d79ae5c8312/volumes" Jul 31 01:06:37.266323 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:37.266238 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7581-140389896667184" Jul 31 01:06:37.266323 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:37.266265 2575 scope.go:117] "RemoveContainer" containerID="85e1e115abbcc13161c89b59b0f25f6f100bde565ba09bb7d79978af9856b4ca" Jul 31 01:06:37.298869 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:37.298831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:37.462587 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:37.462549 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:40.303926 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:40.303890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:40.403434 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:40.403397 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:41.506627 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:41.506592 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:41.732362 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:41.732325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:41.950993 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:41.950955 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:42.054577 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:42.054540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:06:44.480371 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:06:44.480332 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:07:05.332228 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:05.332190 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:06.039253 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.039218 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:06.325036 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.324937 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk"] Jul 31 01:07:06.325544 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.325527 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d12e3b-b948-4524-88a2-4d79ae5c8312" containerName="test-curl-7581-140389896667184" Jul 31 01:07:06.325624 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.325547 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d12e3b-b948-4524-88a2-4d79ae5c8312" containerName="test-curl-7581-140389896667184" Jul 31 01:07:06.325667 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.325653 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d12e3b-b948-4524-88a2-4d79ae5c8312" containerName="test-curl-7581-140389896667184" Jul 31 01:07:06.330785 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.330757 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.333432 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.333407 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0f1466-openshift-default-dockercfg-gw9zq\"" Jul 31 01:07:06.339021 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.338991 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk"] Jul 31 01:07:06.346064 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.346034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:06.408573 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbn5q\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-kube-api-access-bbn5q\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.408806 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-credential-socket\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.408806 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-data\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.408806 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-podinfo\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.408806 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istiod-ca-cert\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.409025 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-token\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.409025 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-certs\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.409025 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-envoy\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.409025 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.408885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-socket\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.509943 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.509899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-data\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.509943 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.509939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-podinfo\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.509961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istiod-ca-cert\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.509984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-token\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-certs\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-envoy\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-socket\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbn5q\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-kube-api-access-bbn5q\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510250 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-credential-socket\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510613 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-data\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510613 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510460 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-certs\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510613 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-socket\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510772 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-credential-socket\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.510964 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.510945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istiod-ca-cert\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.512395 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.512368 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-envoy\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.513033 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.513015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-podinfo\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.517792 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.517758 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-token\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.517913 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.517804 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbn5q\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-kube-api-access-bbn5q\") pod \"e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.563672 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.563632 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:06.645772 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.645672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:06.815972 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.815944 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk"] Jul 31 01:07:06.818290 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:07:06.818264 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28b0edd_b7da_4ebf_9afd_08e899d91ae5.slice/crio-640017a91bfbb480cd9cf7012c7fbbce3b3603bb9686520ebd111ff65d0c35c1 WatchSource:0}: Error finding container 640017a91bfbb480cd9cf7012c7fbbce3b3603bb9686520ebd111ff65d0c35c1: Status 404 returned error can't find the container with id 640017a91bfbb480cd9cf7012c7fbbce3b3603bb9686520ebd111ff65d0c35c1 Jul 31 01:07:06.820520 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.820486 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 01:07:06.820609 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.820560 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 01:07:06.820609 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:06.820589 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 01:07:07.425212 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:07.425172 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" event={"ID":"d28b0edd-b7da-4ebf-9afd-08e899d91ae5","Type":"ContainerStarted","Data":"dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9"} Jul 31 01:07:07.425617 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:07.425228 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" event={"ID":"d28b0edd-b7da-4ebf-9afd-08e899d91ae5","Type":"ContainerStarted","Data":"640017a91bfbb480cd9cf7012c7fbbce3b3603bb9686520ebd111ff65d0c35c1"} Jul 31 01:07:07.444331 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:07.444278 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" podStartSLOduration=1.4442598420000001 podStartE2EDuration="1.444259842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:07:07.442340403 +0000 UTC m=+3636.965308181" watchObservedRunningTime="2026-07-31 01:07:07.444259842 +0000 UTC m=+3636.967227588" Jul 31 01:07:07.646353 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:07.646320 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:07.651481 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:07.651453 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:08.431833 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:08.431793 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:08.432961 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:08.432935 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:09.409510 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:09.409475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:09.510384 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:09.510347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:10.478276 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:10.478241 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:10.668292 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:10.668249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:10.975281 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:10.975246 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:11.070592 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:11.070542 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:14.976336 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:14.976295 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:07:24.474247 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:24.474202 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 31 01:07:24.474709 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:24.474278 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 31 01:07:25.523485 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:25.523397 2575 generic.go:356] "Generic (PLEG): container finished" podID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerID="dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9" exitCode=0 Jul 31 01:07:25.523485 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:25.523470 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" event={"ID":"d28b0edd-b7da-4ebf-9afd-08e899d91ae5","Type":"ContainerDied","Data":"dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9"} Jul 31 01:07:25.523932 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:25.523884 2575 scope.go:117] "RemoveContainer" containerID="dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9" Jul 31 01:07:25.524928 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:25.524899 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 01:07:25.525029 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:25.524989 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 01:07:25.525092 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:25.525039 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 01:07:26.534091 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:26.534051 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" event={"ID":"d28b0edd-b7da-4ebf-9afd-08e899d91ae5","Type":"ContainerStarted","Data":"4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769"} Jul 31 01:07:26.647141 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:26.647074 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:26.647430 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:26.647398 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 31 01:07:26.647610 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:26.647581 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 31 01:07:27.647248 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:27.647206 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 31 01:07:27.647893 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:27.647855 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 31 01:07:28.647081 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:28.647027 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 31 01:07:28.647340 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:28.647105 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 31 01:07:29.647444 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:29.646947 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 31 01:07:29.647444 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:29.647015 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 31 01:07:30.646685 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:30.646625 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 31 01:07:30.647423 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:30.646974 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 31 01:07:31.651835 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:31.651789 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:31.652328 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:31.652304 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:31.653158 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:31.653135 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:07:34.415053 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:34.415009 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:55.477603 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.477510 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc"] Jul 31 01:07:55.481249 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.481224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.485019 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.484977 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2b1008\"/\"ipp-route-2b1008-kserve-self-signed-certs\"" Jul 31 01:07:55.485224 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.485036 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2b1008\"/\"openshift-service-ca.crt\"" Jul 31 01:07:55.485224 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.485054 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2b1008\"/\"default-dockercfg-2md7q\"" Jul 31 01:07:55.485397 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.485266 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2b1008\"/\"kube-root-ca.crt\"" Jul 31 01:07:55.490984 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.490961 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc"] Jul 31 01:07:55.517588 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.517538 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:07:55.569866 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.569815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-tmp-dir\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.569866 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.569865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218257ac-959b-4164-83a8-ba5b12906bc0-tls-certs\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.570166 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.569893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-model-cache\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.570166 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.569928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjsx\" (UniqueName: \"kubernetes.io/projected/218257ac-959b-4164-83a8-ba5b12906bc0-kube-api-access-jkjsx\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.570166 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.570025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-dshm\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.570166 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.570071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-home\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.671493 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.671453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-model-cache\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.671685 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.671506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjsx\" (UniqueName: \"kubernetes.io/projected/218257ac-959b-4164-83a8-ba5b12906bc0-kube-api-access-jkjsx\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.671685 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.671564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-dshm\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.671685 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.671596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-home\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.671685 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.671671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-tmp-dir\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.671905 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.671696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218257ac-959b-4164-83a8-ba5b12906bc0-tls-certs\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.671986 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.671960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-model-cache\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.672072 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.672050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-tmp-dir\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.672154 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.672094 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-home\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.673838 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.673821 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-dshm\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.674180 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.674164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218257ac-959b-4164-83a8-ba5b12906bc0-tls-certs\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.679103 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.679075 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjsx\" (UniqueName: \"kubernetes.io/projected/218257ac-959b-4164-83a8-ba5b12906bc0-kube-api-access-jkjsx\") pod \"ipp-route-2b1008-kserve-685d476dcf-7w4dc\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.793435 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.793331 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:07:55.931672 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:55.931646 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc"] Jul 31 01:07:55.933677 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:07:55.933641 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod218257ac_959b_4164_83a8_ba5b12906bc0.slice/crio-d27d0eefabd3ebfef6886cbc5baf3afd3d52a20ad1f5331c967aec36489cc800 WatchSource:0}: Error finding container d27d0eefabd3ebfef6886cbc5baf3afd3d52a20ad1f5331c967aec36489cc800: Status 404 returned error can't find the container with id d27d0eefabd3ebfef6886cbc5baf3afd3d52a20ad1f5331c967aec36489cc800 Jul 31 01:07:56.695688 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:56.695647 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" event={"ID":"218257ac-959b-4164-83a8-ba5b12906bc0","Type":"ContainerStarted","Data":"57458522fe2391fec61be3aa1c637b66cdc2c7d4caf190a1d9df80f31191e685"} Jul 31 01:07:56.696099 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:56.695697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" event={"ID":"218257ac-959b-4164-83a8-ba5b12906bc0","Type":"ContainerStarted","Data":"d27d0eefabd3ebfef6886cbc5baf3afd3d52a20ad1f5331c967aec36489cc800"} Jul 31 01:07:56.711944 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:56.711888 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" podStartSLOduration=1.711872566 podStartE2EDuration="1.711872566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:07:56.709830242 +0000 UTC m=+3686.232798011" watchObservedRunningTime="2026-07-31 01:07:56.711872566 +0000 UTC m=+3686.234840311" Jul 31 01:07:58.189954 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:07:58.189919 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:08:05.794495 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:05.794449 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:08:05.794951 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:05.794527 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:08:05.807256 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:05.807225 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:08:06.761583 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:06.761550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:08:11.881860 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:11.881820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:20.382821 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:20.382776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:08:31.889497 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:31.889464 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:32.208407 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:32.208372 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:32.375853 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:32.375809 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:34.901978 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:34.901925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:35.006587 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:35.006548 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:36.092903 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:36.092861 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:36.516497 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:36.516461 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:36.616391 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:36.616358 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:36.696608 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:36.696574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:08:39.489394 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:39.489345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:08:59.915360 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:08:59.915319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:06.726078 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:06.726038 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:06.791676 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:06.791625 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:06.964432 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:06.964401 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:07.034869 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:07.034781 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:07.356300 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:07.356210 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:10.376419 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:10.376380 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:09:12.914286 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:12.914248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:12.972519 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:12.972479 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:13.187704 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:13.187615 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:13.422775 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:13.422737 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:30.680969 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:30.680874 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:09:32.530646 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:32.530601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:32.587551 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:32.587514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:32.640076 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:32.640043 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:32.695409 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:32.695373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:33.095041 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:33.095005 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:38.082828 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:38.082790 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc"] Jul 31 01:09:38.122084 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:38.122050 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:09:39.976386 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:39.976347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:09:53.102402 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.102330 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" podUID="218257ac-959b-4164-83a8-ba5b12906bc0" containerName="main" containerID="cri-o://57458522fe2391fec61be3aa1c637b66cdc2c7d4caf190a1d9df80f31191e685" gracePeriod=45 Jul 31 01:09:53.301662 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.301618 2575 generic.go:356] "Generic (PLEG): container finished" podID="218257ac-959b-4164-83a8-ba5b12906bc0" containerID="57458522fe2391fec61be3aa1c637b66cdc2c7d4caf190a1d9df80f31191e685" exitCode=0 Jul 31 01:09:53.301834 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.301688 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" event={"ID":"218257ac-959b-4164-83a8-ba5b12906bc0","Type":"ContainerDied","Data":"57458522fe2391fec61be3aa1c637b66cdc2c7d4caf190a1d9df80f31191e685"} Jul 31 01:09:53.356174 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.356096 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:09:53.375165 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375113 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218257ac-959b-4164-83a8-ba5b12906bc0-tls-certs\") pod \"218257ac-959b-4164-83a8-ba5b12906bc0\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " Jul 31 01:09:53.375337 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375199 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-model-cache\") pod \"218257ac-959b-4164-83a8-ba5b12906bc0\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " Jul 31 01:09:53.375337 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375243 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-home\") pod \"218257ac-959b-4164-83a8-ba5b12906bc0\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " Jul 31 01:09:53.375337 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375298 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-dshm\") pod \"218257ac-959b-4164-83a8-ba5b12906bc0\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " Jul 31 01:09:53.375337 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375333 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkjsx\" (UniqueName: \"kubernetes.io/projected/218257ac-959b-4164-83a8-ba5b12906bc0-kube-api-access-jkjsx\") pod \"218257ac-959b-4164-83a8-ba5b12906bc0\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " Jul 31 01:09:53.375554 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375360 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-tmp-dir\") pod \"218257ac-959b-4164-83a8-ba5b12906bc0\" (UID: \"218257ac-959b-4164-83a8-ba5b12906bc0\") " Jul 31 01:09:53.375554 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375516 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "218257ac-959b-4164-83a8-ba5b12906bc0" (UID: "218257ac-959b-4164-83a8-ba5b12906bc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:09:53.375660 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375599 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-home" (OuterVolumeSpecName: "home") pod "218257ac-959b-4164-83a8-ba5b12906bc0" (UID: "218257ac-959b-4164-83a8-ba5b12906bc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:09:53.375772 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375748 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "218257ac-959b-4164-83a8-ba5b12906bc0" (UID: "218257ac-959b-4164-83a8-ba5b12906bc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:09:53.375930 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.375761 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-model-cache\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:09:53.376032 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.376021 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-home\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:09:53.377834 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.377806 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218257ac-959b-4164-83a8-ba5b12906bc0-kube-api-access-jkjsx" (OuterVolumeSpecName: "kube-api-access-jkjsx") pod "218257ac-959b-4164-83a8-ba5b12906bc0" (UID: "218257ac-959b-4164-83a8-ba5b12906bc0"). InnerVolumeSpecName "kube-api-access-jkjsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:09:53.377949 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.377811 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218257ac-959b-4164-83a8-ba5b12906bc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "218257ac-959b-4164-83a8-ba5b12906bc0" (UID: "218257ac-959b-4164-83a8-ba5b12906bc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 01:09:53.378053 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.378033 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-dshm" (OuterVolumeSpecName: "dshm") pod "218257ac-959b-4164-83a8-ba5b12906bc0" (UID: "218257ac-959b-4164-83a8-ba5b12906bc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:09:53.477010 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.476969 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218257ac-959b-4164-83a8-ba5b12906bc0-tls-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:09:53.477010 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.477003 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-dshm\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:09:53.477010 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.477013 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjsx\" (UniqueName: \"kubernetes.io/projected/218257ac-959b-4164-83a8-ba5b12906bc0-kube-api-access-jkjsx\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:09:53.477360 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:53.477024 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/218257ac-959b-4164-83a8-ba5b12906bc0-tmp-dir\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:09:54.310783 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:54.310752 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" Jul 31 01:09:54.310783 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:54.310764 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc" event={"ID":"218257ac-959b-4164-83a8-ba5b12906bc0","Type":"ContainerDied","Data":"d27d0eefabd3ebfef6886cbc5baf3afd3d52a20ad1f5331c967aec36489cc800"} Jul 31 01:09:54.311334 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:54.310816 2575 scope.go:117] "RemoveContainer" containerID="57458522fe2391fec61be3aa1c637b66cdc2c7d4caf190a1d9df80f31191e685" Jul 31 01:09:54.332266 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:54.332225 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc"] Jul 31 01:09:54.334835 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:54.334808 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2b1008/ipp-route-2b1008-kserve-685d476dcf-7w4dc"] Jul 31 01:09:55.049812 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:09:55.049776 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="218257ac-959b-4164-83a8-ba5b12906bc0" path="/var/lib/kubelet/pods/218257ac-959b-4164-83a8-ba5b12906bc0/volumes" Jul 31 01:10:12.694806 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:12.694758 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:12.745161 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:12.745098 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:14.259000 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:14.258963 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:14.943696 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:14.943665 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:20.438306 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:20.438263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:34.408092 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:34.408055 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:34.509416 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:34.509376 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:34.722515 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:34.722422 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:34.928515 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:34.928483 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:45.486176 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:45.486140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:45.585925 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:45.585885 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:45.651575 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:45.651537 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:45.698974 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:45.698932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:46.020175 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:46.020140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:47.886056 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:47.886017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:10:51.762943 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:51.762883 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:51.818699 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:51.818656 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:52.005958 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:52.005925 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk"] Jul 31 01:10:52.006242 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:52.006207 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" containerID="cri-o://4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769" gracePeriod=30 Jul 31 01:10:52.020550 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:52.020464 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:52.251709 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:52.251673 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:10:57.260675 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.260648 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:10:57.365538 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.365445 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-credential-socket\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.365538 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.365494 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istiod-ca-cert\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.365538 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.365514 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-data\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.365812 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.365548 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbn5q\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-kube-api-access-bbn5q\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.365812 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.365660 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-token\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.365812 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.365707 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-envoy\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.365812 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.365760 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:10:57.366294 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.365752 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-podinfo\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.366542 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.366334 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-socket\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.366542 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.366427 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-certs\") pod \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\" (UID: \"d28b0edd-b7da-4ebf-9afd-08e899d91ae5\") " Jul 31 01:10:57.366542 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.366435 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-data" (OuterVolumeSpecName: "istio-data") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:10:57.367031 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.366859 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:10:57.367031 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.366261 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 01:10:57.368435 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.368177 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-credential-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.368435 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.368213 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istiod-ca-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.368435 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.368228 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-data\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.368435 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.368241 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.368697 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.368449 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:10:57.369490 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.369448 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-kube-api-access-bbn5q" (OuterVolumeSpecName: "kube-api-access-bbn5q") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "kube-api-access-bbn5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:10:57.374969 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.374934 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:10:57.375103 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.375020 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 01:10:57.375103 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.375044 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d28b0edd-b7da-4ebf-9afd-08e899d91ae5" (UID: "d28b0edd-b7da-4ebf-9afd-08e899d91ae5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:10:57.469042 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.469005 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-podinfo\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.469042 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.469036 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-workload-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.469042 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.469047 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbn5q\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-kube-api-access-bbn5q\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.469322 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.469056 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-token\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.469322 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.469065 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28b0edd-b7da-4ebf-9afd-08e899d91ae5-istio-envoy\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:10:57.638306 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.638214 2575 generic.go:356] "Generic (PLEG): container finished" podID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerID="4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769" exitCode=0 Jul 31 01:10:57.638306 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.638278 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" Jul 31 01:10:57.638568 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.638303 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" event={"ID":"d28b0edd-b7da-4ebf-9afd-08e899d91ae5","Type":"ContainerDied","Data":"4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769"} Jul 31 01:10:57.638568 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.638398 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk" event={"ID":"d28b0edd-b7da-4ebf-9afd-08e899d91ae5","Type":"ContainerDied","Data":"640017a91bfbb480cd9cf7012c7fbbce3b3603bb9686520ebd111ff65d0c35c1"} Jul 31 01:10:57.638568 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.638429 2575 scope.go:117] "RemoveContainer" containerID="4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769" Jul 31 01:10:57.648352 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.648331 2575 scope.go:117] "RemoveContainer" containerID="dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9" Jul 31 01:10:57.657903 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.657881 2575 scope.go:117] "RemoveContainer" containerID="4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769" Jul 31 01:10:57.658348 ip-10-0-134-191 kubenswrapper[2575]: E0731 01:10:57.658325 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769\": container with ID starting with 4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769 not found: ID does not exist" containerID="4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769" Jul 31 01:10:57.658438 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.658356 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769"} err="failed to get container status \"4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769\": rpc error: code = NotFound desc = could not find container \"4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769\": container with ID starting with 4501679c9a92d07f200a0f98016be872ebcfc71241c78036f3c7aa366366a769 not found: ID does not exist" Jul 31 01:10:57.658438 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.658376 2575 scope.go:117] "RemoveContainer" containerID="dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9" Jul 31 01:10:57.658682 ip-10-0-134-191 kubenswrapper[2575]: E0731 01:10:57.658662 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9\": container with ID starting with dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9 not found: ID does not exist" containerID="dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9" Jul 31 01:10:57.658726 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.658688 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9"} err="failed to get container status \"dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9\": rpc error: code = NotFound desc = could not find container \"dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9\": container with ID starting with dcf12507550eccca484315c13233afaf53359ff48bdd8eb80e9aa26b24e5a2c9 not found: ID does not exist" Jul 31 01:10:57.660254 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.660233 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk"] Jul 31 01:10:57.663192 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:57.663164 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0f1466-openshift-default-579769bd66-96ktk"] Jul 31 01:10:59.048519 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:10:59.048484 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" path="/var/lib/kubelet/pods/d28b0edd-b7da-4ebf-9afd-08e899d91ae5/volumes" Jul 31 01:11:02.758351 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:02.758313 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:02.818274 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:02.818240 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:02.880208 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:02.880153 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:02.980523 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:02.980477 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:03.036658 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:03.036565 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:03.367057 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:03.366974 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:05.076414 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:05.076373 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:11:09.245802 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:09.245767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:09.346491 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:09.346455 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:09.503740 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:09.503650 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:09.726469 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:09.726428 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:20.295940 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:20.295902 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:20.358033 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:20.358004 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:20.455973 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:20.455912 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:20.510783 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:20.510739 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:20.842568 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:20.842526 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:22.379439 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:22.379399 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6pkb6"] Jul 31 01:11:26.534625 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:26.534584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:26.634854 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:26.634811 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:26.786189 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:26.786076 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4"] Jul 31 01:11:26.786370 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:26.786341 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" containerID="cri-o://c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf" gracePeriod=30 Jul 31 01:11:26.809842 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:26.809801 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:27.050520 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:27.050433 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:32.051398 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.051330 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 01:11:32.097868 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.097834 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-envoy\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098062 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.097899 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-podinfo\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098062 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.097919 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-data\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098062 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.097935 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-certs\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098062 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.097960 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istiod-ca-cert\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098062 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.097988 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkfcz\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-kube-api-access-pkfcz\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098062 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098046 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-token\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098400 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098103 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-credential-socket\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098400 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098152 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-socket\") pod \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\" (UID: \"10bd3c7b-e92a-47b5-a928-363eb1d9bf74\") " Jul 31 01:11:32.098400 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098240 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-data" (OuterVolumeSpecName: "istio-data") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:11:32.098400 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098288 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:11:32.098604 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098435 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:11:32.098604 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098550 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-data\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.098604 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098567 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-certs\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.098604 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098581 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-workload-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.098820 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098667 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:11:32.098965 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.098936 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 01:11:32.100253 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.100225 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 01:11:32.100419 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.100398 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-kube-api-access-pkfcz" (OuterVolumeSpecName: "kube-api-access-pkfcz") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "kube-api-access-pkfcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:11:32.100490 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.100436 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-token" (OuterVolumeSpecName: "istio-token") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 01:11:32.100722 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.100707 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "10bd3c7b-e92a-47b5-a928-363eb1d9bf74" (UID: "10bd3c7b-e92a-47b5-a928-363eb1d9bf74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 01:11:32.199199 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.199164 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-podinfo\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.199199 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.199191 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istiod-ca-cert\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.199199 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.199201 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkfcz\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-kube-api-access-pkfcz\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.199493 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.199211 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-token\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.199493 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.199222 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-credential-socket\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.199493 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.199230 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bd3c7b-e92a-47b5-a928-363eb1d9bf74-istio-envoy\") on node \"ip-10-0-134-191.ec2.internal\" DevicePath \"\"" Jul 31 01:11:32.816404 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.816366 2575 generic.go:356] "Generic (PLEG): container finished" podID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerID="c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf" exitCode=0 Jul 31 01:11:32.816589 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.816422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" event={"ID":"10bd3c7b-e92a-47b5-a928-363eb1d9bf74","Type":"ContainerDied","Data":"c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf"} Jul 31 01:11:32.816589 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.816425 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" Jul 31 01:11:32.816589 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.816489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4" event={"ID":"10bd3c7b-e92a-47b5-a928-363eb1d9bf74","Type":"ContainerDied","Data":"84b7a8ab73a103f86c975379d6da94085fe6a09b94831d909625e6e3b87871d7"} Jul 31 01:11:32.816589 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.816510 2575 scope.go:117] "RemoveContainer" containerID="c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf" Jul 31 01:11:32.828052 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.827990 2575 scope.go:117] "RemoveContainer" containerID="0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c" Jul 31 01:11:32.836825 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.836801 2575 scope.go:117] "RemoveContainer" containerID="c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf" Jul 31 01:11:32.837180 ip-10-0-134-191 kubenswrapper[2575]: E0731 01:11:32.837154 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf\": container with ID starting with c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf not found: ID does not exist" containerID="c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf" Jul 31 01:11:32.837246 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.837195 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf"} err="failed to get container status \"c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf\": rpc error: code = NotFound desc = could not find container \"c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf\": container with ID starting with c9b1d77eb79ad0e4f70ed702ab38ec6d5ccf925811136b64dcf3cb0487f2afdf not found: ID does not exist" Jul 31 01:11:32.837246 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.837222 2575 scope.go:117] "RemoveContainer" containerID="0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c" Jul 31 01:11:32.837510 ip-10-0-134-191 kubenswrapper[2575]: E0731 01:11:32.837491 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c\": container with ID starting with 0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c not found: ID does not exist" containerID="0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c" Jul 31 01:11:32.837557 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.837517 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c"} err="failed to get container status \"0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c\": rpc error: code = NotFound desc = could not find container \"0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c\": container with ID starting with 0213b0a814e9c02fdaeb0b300f996484ed8f7cbc400b2ee2b0ed506b07255a0c not found: ID does not exist" Jul 31 01:11:32.838767 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.838713 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4"] Jul 31 01:11:32.841532 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:32.841507 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b32567-openshift-default-686684fc4d-dn2h4"] Jul 31 01:11:33.048862 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:33.048829 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" path="/var/lib/kubelet/pods/10bd3c7b-e92a-47b5-a928-363eb1d9bf74/volumes" Jul 31 01:11:33.191876 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:33.191841 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-79ms7_003e59a8-eb50-4c41-b4fb-ea86b9d87b5d/manager/0.log" Jul 31 01:11:33.323199 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:33.323170 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tb58t_17f3ca15-7db5-4bd9-a057-ca804bfcfcce/manager/0.log" Jul 31 01:11:33.565380 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:33.565294 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vh2xp_8bfac874-6ea0-49f2-a296-6e2557d16606/server/0.log" Jul 31 01:11:33.709334 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:33.709302 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xqvts_bab7c36f-d5a6-43ed-80ad-ceec181024ab/manager/0.log" Jul 31 01:11:33.828649 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:33.828572 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff98467-4m269_af0b51d3-ea77-4ded-b863-e3a2f62dcd3c/copy-manifests/0.log" Jul 31 01:11:33.846295 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:33.846260 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff98467-4m269_af0b51d3-ea77-4ded-b863-e3a2f62dcd3c/manager/0.log" Jul 31 01:11:34.723234 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:34.723206 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29757645-dpzw8_061304a2-259a-40f6-978e-4b73a303eef7/cleanup/0.log" Jul 31 01:11:35.889497 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:35.889458 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-28n2g_242f44ec-cc86-4923-bf5a-70aa6a437839/manager/0.log" Jul 31 01:11:36.008016 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:36.007985 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-qxpsf_b488704f-6e0c-434c-8fa5-22500a86c020/manager/0.log" Jul 31 01:11:36.120734 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:36.120705 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-kk5lv_54239473-e042-4670-a5b7-ca4cdcb1ad20/kuadrant-console-plugin/0.log" Jul 31 01:11:36.252137 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:36.252087 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-fx69q_7ea8e0db-38b6-4151-b43d-751c831cecd6/manager/0.log" Jul 31 01:11:36.365407 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:36.365378 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-6pkb6_c8d2e371-1816-448e-96e2-76f25edebe39/limitador/0.log" Jul 31 01:11:36.494359 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:36.494325 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-dkxk5_d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe/manager/0.log" Jul 31 01:11:36.874730 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:36.874697 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7_6d8018ac-2096-47a5-8bdd-af717ab2b8c6/istio-proxy/0.log" Jul 31 01:11:37.264163 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:37.264112 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f6dcbc74-vjc79_a32783c1-3abc-4735-9a8f-4bd705d05c20/kube-auth-proxy/0.log" Jul 31 01:11:38.592360 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:38.592327 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4pkv2_103658b2-0dba-42fe-b0a5-fdc1aa8735de/main/0.log" Jul 31 01:11:38.708401 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:38.708370 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6n2x5_a3aebfc2-17ae-4299-982b-c154479165d0/main/0.log" Jul 31 01:11:44.418824 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.418788 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sj55h/must-gather-ng2gj"] Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419155 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="218257ac-959b-4164-83a8-ba5b12906bc0" containerName="main" Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419167 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="218257ac-959b-4164-83a8-ba5b12906bc0" containerName="main" Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419178 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419184 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419193 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419198 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419203 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419209 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" Jul 31 01:11:44.419221 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419223 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" Jul 31 01:11:44.419524 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419228 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" Jul 31 01:11:44.419524 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419283 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" Jul 31 01:11:44.419524 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419290 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" Jul 31 01:11:44.419524 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419295 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="218257ac-959b-4164-83a8-ba5b12906bc0" containerName="main" Jul 31 01:11:44.419524 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419302 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="10bd3c7b-e92a-47b5-a928-363eb1d9bf74" containerName="istio-proxy" Jul 31 01:11:44.419524 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.419394 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d28b0edd-b7da-4ebf-9afd-08e899d91ae5" containerName="istio-proxy" Jul 31 01:11:44.423951 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.423928 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sj55h/must-gather-ng2gj" Jul 31 01:11:44.426716 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.426696 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sj55h\"/\"kube-root-ca.crt\"" Jul 31 01:11:44.427802 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.427780 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sj55h\"/\"default-dockercfg-jj2mk\"" Jul 31 01:11:44.427931 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.427805 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sj55h\"/\"openshift-service-ca.crt\"" Jul 31 01:11:44.430876 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.430849 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sj55h/must-gather-ng2gj"] Jul 31 01:11:44.463949 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.463918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:44.516036 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.515992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90a358a7-6a6a-408a-9321-54133c4ef4b5-must-gather-output\") pod \"must-gather-ng2gj\" (UID: \"90a358a7-6a6a-408a-9321-54133c4ef4b5\") " pod="openshift-must-gather-sj55h/must-gather-ng2gj" Jul 31 01:11:44.516272 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.516174 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnsj\" (UniqueName: \"kubernetes.io/projected/90a358a7-6a6a-408a-9321-54133c4ef4b5-kube-api-access-mdnsj\") pod \"must-gather-ng2gj\" (UID: \"90a358a7-6a6a-408a-9321-54133c4ef4b5\") " pod="openshift-must-gather-sj55h/must-gather-ng2gj" Jul 31 01:11:44.564229 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.564188 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:11:44.617556 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.617518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnsj\" (UniqueName: \"kubernetes.io/projected/90a358a7-6a6a-408a-9321-54133c4ef4b5-kube-api-access-mdnsj\") pod \"must-gather-ng2gj\" (UID: \"90a358a7-6a6a-408a-9321-54133c4ef4b5\") " pod="openshift-must-gather-sj55h/must-gather-ng2gj" Jul 31 01:11:44.617769 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.617572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90a358a7-6a6a-408a-9321-54133c4ef4b5-must-gather-output\") pod \"must-gather-ng2gj\" (UID: \"90a358a7-6a6a-408a-9321-54133c4ef4b5\") " pod="openshift-must-gather-sj55h/must-gather-ng2gj" Jul 31 01:11:44.617885 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.617868 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90a358a7-6a6a-408a-9321-54133c4ef4b5-must-gather-output\") pod \"must-gather-ng2gj\" (UID: \"90a358a7-6a6a-408a-9321-54133c4ef4b5\") " pod="openshift-must-gather-sj55h/must-gather-ng2gj" Jul 31 01:11:44.625259 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.625235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnsj\" (UniqueName: \"kubernetes.io/projected/90a358a7-6a6a-408a-9321-54133c4ef4b5-kube-api-access-mdnsj\") pod \"must-gather-ng2gj\" (UID: \"90a358a7-6a6a-408a-9321-54133c4ef4b5\") " pod="openshift-must-gather-sj55h/must-gather-ng2gj" Jul 31 01:11:44.734288 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.734194 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sj55h/must-gather-ng2gj" Jul 31 01:11:44.868442 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.868414 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sj55h/must-gather-ng2gj"] Jul 31 01:11:44.870871 ip-10-0-134-191 kubenswrapper[2575]: W0731 01:11:44.870844 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90a358a7_6a6a_408a_9321_54133c4ef4b5.slice/crio-b5ba91d59c0bee566f3892c423fb0da28f17df6813d5f1c7ce29550f1b02f27b WatchSource:0}: Error finding container b5ba91d59c0bee566f3892c423fb0da28f17df6813d5f1c7ce29550f1b02f27b: Status 404 returned error can't find the container with id b5ba91d59c0bee566f3892c423fb0da28f17df6813d5f1c7ce29550f1b02f27b Jul 31 01:11:44.873257 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:44.873233 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 01:11:45.883330 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:45.883291 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sj55h/must-gather-ng2gj" event={"ID":"90a358a7-6a6a-408a-9321-54133c4ef4b5","Type":"ContainerStarted","Data":"38ad93f884de1183176ceb4c72313415c72dcabe3485d68977fd69bfa6d12e93"} Jul 31 01:11:45.883785 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:45.883348 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sj55h/must-gather-ng2gj" event={"ID":"90a358a7-6a6a-408a-9321-54133c4ef4b5","Type":"ContainerStarted","Data":"b5ba91d59c0bee566f3892c423fb0da28f17df6813d5f1c7ce29550f1b02f27b"} Jul 31 01:11:46.891382 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:46.891347 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sj55h/must-gather-ng2gj" event={"ID":"90a358a7-6a6a-408a-9321-54133c4ef4b5","Type":"ContainerStarted","Data":"f2eed492038c587b11b7776139e6176994b5d52833c62c838ecd6c6496c66ef2"} Jul 31 01:11:46.906311 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:46.906245 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sj55h/must-gather-ng2gj" podStartSLOduration=2.036732558 podStartE2EDuration="2.906222434s" totalImagesPullingTime="869.489876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 01:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:11:46.90556531 +0000 UTC m=+3916.428533059" watchObservedRunningTime="2026-07-31 01:11:46.906222434 +0000 UTC m=+3916.429190180" Jul 31 01:11:47.310802 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:47.310770 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gjqf5_e3f6ffa3-e0a4-49f0-bfe6-7ca019be77b9/global-pull-secret-syncer/0.log" Jul 31 01:11:47.409488 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:47.409452 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" Jul 31 01:11:47.525022 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:47.524989 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-191.ec2.internal_3820b175871ebd792360ac3026b04d32/haproxy/0.log" Jul 31 01:11:52.521424 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:52.521379 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-28n2g_242f44ec-cc86-4923-bf5a-70aa6a437839/manager/0.log" Jul 31 01:11:52.545684 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:52.545648 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-qxpsf_b488704f-6e0c-434c-8fa5-22500a86c020/manager/0.log" Jul 31 01:11:52.571986 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:52.571957 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-kk5lv_54239473-e042-4670-a5b7-ca4cdcb1ad20/kuadrant-console-plugin/0.log" Jul 31 01:11:52.706578 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:52.706536 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-fx69q_7ea8e0db-38b6-4151-b43d-751c831cecd6/manager/0.log" Jul 31 01:11:52.736580 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:52.736539 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-6pkb6_c8d2e371-1816-448e-96e2-76f25edebe39/limitador/0.log" Jul 31 01:11:52.836733 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:52.836646 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-dkxk5_d53807b2-f7a6-4bd0-a7cd-b6ddc019bdfe/manager/0.log" Jul 31 01:11:54.523585 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:54.523558 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f65b5c676-46bnn_11d84d86-70c6-45c9-a3c4-aabaed72be9d/metrics-server/0.log" Jul 31 01:11:54.563140 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:54.563092 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-xsv8p_71ba5396-440c-4776-9dc6-c22345849b06/monitoring-plugin/0.log" Jul 31 01:11:54.746982 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:54.746954 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9qjm_90c20e03-2d27-4fd7-b91a-0ab366cfcdcc/node-exporter/0.log" Jul 31 01:11:54.766750 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:54.766690 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9qjm_90c20e03-2d27-4fd7-b91a-0ab366cfcdcc/kube-rbac-proxy/0.log" Jul 31 01:11:54.795795 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:54.795719 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9qjm_90c20e03-2d27-4fd7-b91a-0ab366cfcdcc/init-textfile/0.log" Jul 31 01:11:54.822395 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:54.822366 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-f9mc8_4f6e14a0-b26d-4d2a-904c-f78cd4179cae/kube-rbac-proxy-main/0.log" Jul 31 01:11:54.842460 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:54.842429 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-f9mc8_4f6e14a0-b26d-4d2a-904c-f78cd4179cae/kube-rbac-proxy-self/0.log" Jul 31 01:11:54.886312 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:54.886278 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-f9mc8_4f6e14a0-b26d-4d2a-904c-f78cd4179cae/openshift-state-metrics/0.log" Jul 31 01:11:58.051373 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.051339 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f6b47b856-hrcr4_81610e93-bae2-4ce2-82b9-345daebeee94/console/0.log" Jul 31 01:11:58.131503 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.131449 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828"] Jul 31 01:11:58.139981 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.139948 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.142943 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.142912 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828"] Jul 31 01:11:58.173500 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.173410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-podres\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.173500 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.173461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-lib-modules\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.173500 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.173493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6c6h\" (UniqueName: \"kubernetes.io/projected/e9d01316-8da4-48d2-b43e-bee1ce27256e-kube-api-access-n6c6h\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.173797 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.173620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-sys\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.173797 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.173646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-proc\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.274732 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.274694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-podres\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.274732 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.274740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-lib-modules\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.274994 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.274849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6c6h\" (UniqueName: \"kubernetes.io/projected/e9d01316-8da4-48d2-b43e-bee1ce27256e-kube-api-access-n6c6h\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.274994 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.274887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-podres\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.274994 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.274887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-lib-modules\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.275202 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.274996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-sys\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.275202 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.275018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-proc\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.275202 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.275079 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-sys\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.275202 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.275098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9d01316-8da4-48d2-b43e-bee1ce27256e-proc\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.289752 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.289715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6c6h\" (UniqueName: \"kubernetes.io/projected/e9d01316-8da4-48d2-b43e-bee1ce27256e-kube-api-access-n6c6h\") pod \"perf-node-gather-daemonset-dr828\" (UID: \"e9d01316-8da4-48d2-b43e-bee1ce27256e\") " pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.455421 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.455376 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:58.646056 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.646017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828"] Jul 31 01:11:58.991172 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.991076 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" event={"ID":"e9d01316-8da4-48d2-b43e-bee1ce27256e","Type":"ContainerStarted","Data":"240f26f611f3a24732bd784d54fcec2fc33fc6c9619f89a9496527d9326ffcc8"} Jul 31 01:11:58.991172 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.991144 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" event={"ID":"e9d01316-8da4-48d2-b43e-bee1ce27256e","Type":"ContainerStarted","Data":"e020b21a9577ec7c379ce9a8a7e478785bfb387436f56b015ac87771f16e3e70"} Jul 31 01:11:58.991420 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:58.991179 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:11:59.007405 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:11:59.007345 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" podStartSLOduration=1.00731976 podStartE2EDuration="1.00731976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 01:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 01:11:59.005680789 +0000 UTC m=+3928.528648553" watchObservedRunningTime="2026-07-31 01:11:59.00731976 +0000 UTC m=+3928.530287505" Jul 31 01:12:00.037260 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:00.037221 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gtc62_7aa00574-87fc-449d-a891-4b43e2f0d014/dns/0.log" Jul 31 01:12:00.055253 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:00.055222 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gtc62_7aa00574-87fc-449d-a891-4b43e2f0d014/kube-rbac-proxy/0.log" Jul 31 01:12:00.123324 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:00.123297 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5f6lz_814f97fa-4c99-483b-a086-c3da15df60a6/dns-node-resolver/0.log" Jul 31 01:12:00.737605 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:00.737559 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ck7r_232e2b98-3c4e-415e-b538-6e350a6df518/node-ca/0.log" Jul 31 01:12:01.746465 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:01.746432 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7_6d8018ac-2096-47a5-8bdd-af717ab2b8c6/istio-proxy/0.log" Jul 31 01:12:02.112164 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:02.112054 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f6dcbc74-vjc79_a32783c1-3abc-4735-9a8f-4bd705d05c20/kube-auth-proxy/0.log" Jul 31 01:12:03.024190 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:03.024149 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87299_63977669-cfbc-4363-9091-f32b6585ae23/serve-healthcheck-canary/0.log" Jul 31 01:12:03.539697 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:03.539667 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r74xz_31f3ae79-4f9e-46cd-bd66-25e2179d6915/kube-rbac-proxy/0.log" Jul 31 01:12:03.557713 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:03.557684 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r74xz_31f3ae79-4f9e-46cd-bd66-25e2179d6915/exporter/0.log" Jul 31 01:12:03.576972 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:03.576946 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r74xz_31f3ae79-4f9e-46cd-bd66-25e2179d6915/extractor/0.log" Jul 31 01:12:05.008283 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:05.008251 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sj55h/perf-node-gather-daemonset-dr828" Jul 31 01:12:05.793288 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:05.793253 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-79ms7_003e59a8-eb50-4c41-b4fb-ea86b9d87b5d/manager/0.log" Jul 31 01:12:06.276929 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:06.276876 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tb58t_17f3ca15-7db5-4bd9-a057-ca804bfcfcce/manager/0.log" Jul 31 01:12:06.454428 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:06.454397 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vh2xp_8bfac874-6ea0-49f2-a296-6e2557d16606/server/0.log" Jul 31 01:12:06.574784 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:06.574711 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xqvts_bab7c36f-d5a6-43ed-80ad-ceec181024ab/manager/0.log" Jul 31 01:12:06.730845 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:06.730815 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff98467-4m269_af0b51d3-ea77-4ded-b863-e3a2f62dcd3c/manager/0.log" Jul 31 01:12:06.753012 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:06.752985 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff98467-4m269_af0b51d3-ea77-4ded-b863-e3a2f62dcd3c/copy-manifests/0.log" Jul 31 01:12:09.004796 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:09.004763 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7_6d8018ac-2096-47a5-8bdd-af717ab2b8c6/istio-proxy/0.log" Jul 31 01:12:09.340814 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:09.340711 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f6dcbc74-vjc79_a32783c1-3abc-4735-9a8f-4bd705d05c20/kube-auth-proxy/0.log" Jul 31 01:12:10.846338 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:10.846304 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vzdq7_6d8018ac-2096-47a5-8bdd-af717ab2b8c6/istio-proxy/0.log" Jul 31 01:12:11.197897 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:11.197863 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f6dcbc74-vjc79_a32783c1-3abc-4735-9a8f-4bd705d05c20/kube-auth-proxy/0.log" Jul 31 01:12:14.000897 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:14.000858 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cvnn_afbe38cc-74f5-4daf-a158-120a7a583821/konnectivity-agent/0.log" limit=1024 Jul 31 01:12:15.057856 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.057823 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtk76_1f877c84-ab31-4503-9fcd-bb0f621e4ef9/kube-multus-additional-cni-plugins/0.log" Jul 31 01:12:15.081976 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.081947 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtk76_1f877c84-ab31-4503-9fcd-bb0f621e4ef9/egress-router-binary-copy/0.log" Jul 31 01:12:15.099352 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.099324 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtk76_1f877c84-ab31-4503-9fcd-bb0f621e4ef9/cni-plugins/0.log" Jul 31 01:12:15.117967 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.117938 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtk76_1f877c84-ab31-4503-9fcd-bb0f621e4ef9/bond-cni-plugin/0.log" Jul 31 01:12:15.136253 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.136210 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtk76_1f877c84-ab31-4503-9fcd-bb0f621e4ef9/routeoverride-cni/0.log" Jul 31 01:12:15.155767 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.155737 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtk76_1f877c84-ab31-4503-9fcd-bb0f621e4ef9/whereabouts-cni-bincopy/0.log" Jul 31 01:12:15.173487 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.173457 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtk76_1f877c84-ab31-4503-9fcd-bb0f621e4ef9/whereabouts-cni/0.log" Jul 31 01:12:15.292457 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.292427 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wt2nl_fccb5b35-15ad-426c-a0c5-83eb68df8166/kube-multus/0.log" Jul 31 01:12:15.317830 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.317738 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6xpcj_7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d/network-metrics-daemon/0.log" Jul 31 01:12:15.332050 ip-10-0-134-191 kubenswrapper[2575]: I0731 01:12:15.332019 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6xpcj_7156ea2e-a762-4d4b-a0ec-1d70e1d47e0d/kube-rbac-proxy/0.log"