Jul 22 01:34:59.700309 ip-10-0-130-174 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 01:34:59.700320 ip-10-0-130-174 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 01:34:59.700326 ip-10-0-130-174 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 01:34:59.700532 ip-10-0-130-174 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 01:35:09.810058 ip-10-0-130-174 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 01:35:09.810073 ip-10-0-130-174 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5ad245088c834be285406f6cca2fbc46 -- Jul 22 01:37:05.412518 ip-10-0-130-174 systemd[1]: Starting Kubernetes Kubelet... Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.701492 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706550 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706565 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706570 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706581 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706585 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706588 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706591 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:05.801728 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706594 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:05.768242 ip-10-0-130-174 systemd[1]: Started Kubernetes Kubelet. Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706597 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706600 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706603 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706607 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706610 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706613 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706616 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706619 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706622 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706625 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706628 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706630 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706633 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706636 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706638 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706641 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706644 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706648 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:05.803388 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706652 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706655 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706658 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706661 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706664 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706666 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706672 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706675 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706678 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706682 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706686 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706689 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706692 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706695 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706697 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706700 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706702 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706705 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706708 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:05.809994 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706710 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706714 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706717 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706719 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706722 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706725 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706728 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706731 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706733 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706736 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706738 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706741 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706743 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706746 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706748 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706751 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706753 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706756 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706759 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706761 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:05.811003 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706763 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706766 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706768 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706771 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706773 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706776 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706779 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706782 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706786 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706789 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706791 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706794 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706796 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706799 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706803 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706805 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706808 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706810 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706814 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706816 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:05.823240 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706819 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706821 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706824 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706827 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706829 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706832 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706834 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706836 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706839 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706842 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706844 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706847 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706849 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.706851 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708061 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708069 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708072 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708075 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708078 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708081 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708084 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:05.824275 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708086 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708089 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708092 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708096 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708099 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708101 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708104 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708106 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708109 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708111 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708114 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708117 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708119 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708122 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708124 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708127 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708129 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708132 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708134 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708137 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:05.825307 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708140 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708143 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708146 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708148 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708151 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708153 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708156 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708159 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708161 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708164 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708166 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708169 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708171 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708174 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708177 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708179 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708182 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708184 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708186 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708189 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:05.921651 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708195 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708197 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708200 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708205 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708208 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708227 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708230 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708233 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708235 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708238 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708241 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708245 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708247 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708250 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708253 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708255 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708258 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708262 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708265 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:05.923028 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708268 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708271 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708274 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708277 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708279 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708283 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708286 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708288 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708292 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708294 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708297 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708300 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708303 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708307 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708310 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708312 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708315 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708318 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708321 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708323 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:05.924058 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708326 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708328 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708331 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708335 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708337 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708340 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708344 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708348 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708351 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708354 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708356 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708359 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708363 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708725 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708730 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708733 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708736 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708739 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708741 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:05.925069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708744 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708747 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708750 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708752 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708755 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708758 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708762 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708764 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708768 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708771 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708774 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708777 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708780 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708783 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708785 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708789 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708793 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708796 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708799 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:05.926094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708801 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708804 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708806 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708809 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708811 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708814 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708816 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708818 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708821 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708824 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708826 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708829 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708831 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708834 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708836 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708839 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708841 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708844 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708846 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708850 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:05.927103 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708852 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708856 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708859 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708861 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708864 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708866 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708869 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708872 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708874 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708876 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708879 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708881 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708884 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708889 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708892 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708895 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708898 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708901 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708905 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:05.928176 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708908 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708911 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708914 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708917 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708919 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708922 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708924 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708927 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708929 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708932 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708934 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708937 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708941 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708943 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708947 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708949 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708952 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708954 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708957 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708960 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:05.929146 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708962 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708965 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708968 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708970 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708973 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708976 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708978 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708980 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708983 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708985 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708987 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708990 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708992 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708995 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.708997 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709078 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709088 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709104 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709110 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709116 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709120 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 01:37:05.930169 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709124 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709128 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709131 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709135 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709139 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709144 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709147 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709149 2570 flags.go:64] FLAG: --cgroup-root="" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709152 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709155 2570 flags.go:64] FLAG: --client-ca-file="" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709158 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709161 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709165 2570 flags.go:64] FLAG: --cluster-domain="" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709168 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709171 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709174 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709177 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709181 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709184 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709187 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709190 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709193 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709196 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709199 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 01:37:05.931189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709202 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709205 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709209 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709227 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709230 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709233 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709240 2570 flags.go:64] FLAG: --enable-server="true" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709244 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709249 2570 flags.go:64] FLAG: --event-burst="100" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709252 2570 flags.go:64] FLAG: --event-qps="50" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709255 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709259 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709262 2570 flags.go:64] FLAG: --eviction-hard="" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709268 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709271 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709274 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709277 2570 flags.go:64] FLAG: --eviction-soft="" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709279 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709282 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709286 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709290 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709293 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709296 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709298 2570 flags.go:64] FLAG: --feature-gates="" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709302 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 01:37:05.932243 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709305 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709308 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709312 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709314 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709318 2570 flags.go:64] FLAG: --help="false" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709320 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-130-174.ec2.internal" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709323 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709326 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709329 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709333 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709336 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709339 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709342 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709344 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709347 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709350 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709353 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709356 2570 flags.go:64] FLAG: --kube-reserved="" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709360 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709363 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709367 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709370 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709373 2570 flags.go:64] FLAG: --lock-file="" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709376 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 01:37:05.933300 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709379 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709382 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709387 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709390 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709392 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709395 2570 flags.go:64] FLAG: --logging-format="text" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709398 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709401 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709404 2570 flags.go:64] FLAG: --manifest-url="" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709407 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709411 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709413 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709417 2570 flags.go:64] FLAG: --max-pods="110" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709420 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709423 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709426 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709428 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709431 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709434 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709437 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709444 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709447 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709450 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 01:37:05.934339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709453 2570 flags.go:64] FLAG: --pod-cidr="" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709455 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709460 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709463 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709468 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709472 2570 flags.go:64] FLAG: --port="10250" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709475 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709478 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e46299d08791584" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709481 2570 flags.go:64] FLAG: --qos-reserved="" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709484 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709487 2570 flags.go:64] FLAG: --register-node="true" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709490 2570 flags.go:64] FLAG: --register-with-taints="" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709494 2570 flags.go:64] FLAG: --registry-burst="10" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709497 2570 flags.go:64] FLAG: --registry-qps="5" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709500 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709503 2570 flags.go:64] FLAG: --reserved-memory="" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709507 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709510 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709513 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709516 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709518 2570 flags.go:64] FLAG: --runonce="false" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709521 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709524 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709527 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709530 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 01:37:05.935428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709533 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709535 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709538 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709541 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709544 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709547 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709549 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709552 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709555 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709558 2570 flags.go:64] FLAG: --system-cgroups="" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709561 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709567 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709572 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709575 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709578 2570 flags.go:64] FLAG: --tls-min-version="" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709581 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709584 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709587 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709590 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709593 2570 flags.go:64] FLAG: --v="2" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709597 2570 flags.go:64] FLAG: --version="false" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709601 2570 flags.go:64] FLAG: --vmodule="" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709604 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.709607 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709716 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:05.936157 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709720 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709725 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709728 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709731 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709734 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709736 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709739 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709741 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709743 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709746 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709748 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709751 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709754 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709756 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709759 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709761 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709764 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709766 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709771 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:05.936947 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709775 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709779 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709782 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709784 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709787 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709790 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709793 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709796 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709798 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709801 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709804 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709807 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709809 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709812 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709814 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709817 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709819 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709822 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709824 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:05.937766 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709826 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709829 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709831 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709834 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709837 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709839 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709841 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709844 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709846 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709849 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709851 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709854 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709858 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709862 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709865 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709867 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709869 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709872 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709875 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709877 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:05.942556 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709879 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709882 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709884 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709887 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709890 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709892 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709894 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709897 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709899 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709902 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709906 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709910 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709913 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709917 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709920 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709923 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709925 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709928 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709930 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709933 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:05.943338 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709935 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709937 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709940 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709942 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709947 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709951 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709953 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709956 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709958 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709962 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709965 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709968 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709971 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709973 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709976 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709979 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709981 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709985 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709987 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:06.049623 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.709990 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.710468 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.717626 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.717645 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.718207 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.718229 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718306 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718314 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718319 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718324 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718329 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718333 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718339 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718346 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:06.050425 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718351 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718355 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718359 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718364 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718368 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718372 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718377 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718381 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718384 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718389 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718393 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718397 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718400 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718404 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718408 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718412 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718416 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718420 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718424 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718429 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:06.051163 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718433 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718438 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718442 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718445 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718449 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718453 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718457 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718463 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718468 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718472 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718477 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718481 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718485 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718490 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718493 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718498 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718502 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718506 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718510 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:06.052186 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718513 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718517 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718521 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718525 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718529 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718533 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718537 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718541 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718545 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718549 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718554 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718557 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718561 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718566 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718570 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718581 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718591 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718599 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718603 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718607 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:06.053065 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718611 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718615 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718619 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718623 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718626 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718632 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718637 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718641 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718645 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718649 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718653 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718657 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718660 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718664 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718668 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718672 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718676 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718679 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718684 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:06.053907 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718687 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718691 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718695 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718699 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718703 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718707 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718710 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718714 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718718 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718724 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718729 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718733 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718738 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.718745 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718931 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:06.054765 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718940 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718944 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718948 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718952 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718956 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718960 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718964 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718969 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718972 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718976 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718980 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718984 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718988 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718993 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.718996 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719000 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719004 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719008 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719012 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:06.055633 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719015 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719021 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719026 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719030 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719035 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719039 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719043 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719049 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719054 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719059 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719064 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719067 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719071 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719075 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719080 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719085 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719089 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719093 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719097 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:06.056509 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719101 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719105 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719108 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719112 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719116 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719120 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719124 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719128 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719131 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719135 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719139 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719143 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719146 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719150 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719154 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719158 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719162 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719166 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719170 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:06.057375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719174 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719180 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719183 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719189 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719193 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719197 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719201 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719204 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719208 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719226 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719230 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719234 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719237 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719241 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719245 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719249 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719253 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719258 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719262 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719265 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:06.058239 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719269 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719273 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719279 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719284 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719289 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719293 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719297 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719301 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719305 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719309 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719313 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719316 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719321 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719325 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719329 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719335 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719340 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719344 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719349 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719353 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:06.059094 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:05.719356 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.719363 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.719553 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.722353 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.723465 2570 server.go:1019] "Starting client certificate rotation" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.723560 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.723589 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.726179 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.727772 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.736257 2570 log.go:25] "Validated CRI v1 runtime API" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.740243 2570 log.go:25] "Validated CRI v1 image API" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.741031 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 01:37:06.059978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.742941 2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cabe8e8f-2896-45e4-ba87-35f9c6aeaf79:/dev/nvme0n1p4 edb595dc-80c4-4235-a765-d324a01ccf2d:/dev/nvme0n1p3] Jul 22 01:37:06.060413 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.742956 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 01:37:06.060826 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.748235 2570 manager.go:216] Machine: {Timestamp:2026-07-22 01:37:05.746567405 +0000 UTC m=+0.244008773 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105322 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a288ef94eafa185194d13351b7a85 SystemUUID:ec2a288e-f94e-afa1-8519-4d13351b7a85 BootID:5ad24508-8c83-4be2-8540-6f6cca2fbc46 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c9:d9:e2:27:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c9:d9:e2:27:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:f7:23:2a:46:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 01:37:06.060826 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.748332 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 01:37:06.060826 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.748407 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 01:37:06.060826 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.750863 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 01:37:06.060826 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.750897 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-174.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 01:37:06.060826 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.751535 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 01:37:06.060826 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.751550 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 01:37:06.060826 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.751561 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.751610 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.752069 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.753241 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757031 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757169 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757202 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757241 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757266 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757284 2570 kubelet.go:398] "Adding apiserver pod source" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757313 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757703 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.757723 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.759449 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.761417 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.761462 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762510 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762527 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762538 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762549 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762568 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762584 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762596 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762604 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762617 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762626 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762637 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.762645 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.763226 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.763278 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.767318 2570 server.go:1268] "Started kubelet" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.767398 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.767414 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.767470 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.768889 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.769537 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-174.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.769621 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.770102 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-174.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.770856 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 22 01:37:06.061250 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.773229 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.775624 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-174.ec2.internal.18c478d90116c025 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-174.ec2.internal,UID:ip-10-0-130-174.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-174.ec2.internal,},FirstTimestamp:2026-07-22 01:37:05.767063589 +0000 UTC m=+0.264504938,LastTimestamp:2026-07-22 01:37:05.767063589 +0000 UTC m=+0.264504938,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-174.ec2.internal,}" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.777375 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.777387 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.777491 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.777984 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778092 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778107 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778251 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778259 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778858 2570 factory.go:153] Registering CRI-O factory Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778888 2570 factory.go:223] Registration of the crio container factory successfully Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778886 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lpd6b" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778940 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778958 2570 factory.go:55] Registering systemd factory Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778965 2570 factory.go:223] Registration of the systemd container factory successfully Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778983 2570 factory.go:103] Registering Raw factory Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.778995 2570 manager.go:1196] Started watching for new ooms in manager Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.779374 2570 manager.go:349] Starting recovery of all containers Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.780557 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.780588 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-174.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.785072 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lpd6b" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.785651 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.790059 2570 manager.go:354] Recovery completed Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.794557 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.796328 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.796360 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.796372 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.796878 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.796889 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.796908 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.796879 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.797610 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:06.062558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.797632 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:06.063869 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.797644 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:06.063869 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.798403 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-174.ec2.internal.18c478d902d5807f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-174.ec2.internal,UID:ip-10-0-130-174.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-174.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-174.ec2.internal,},FirstTimestamp:2026-07-22 01:37:05.796341887 +0000 UTC m=+0.293783242,LastTimestamp:2026-07-22 01:37:05.796341887 +0000 UTC m=+0.293783242,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-174.ec2.internal,}" Jul 22 01:37:06.063869 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.798979 2570 policy_none.go:49] "None policy: Start" Jul 22 01:37:06.063869 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.798991 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 01:37:06.063869 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.799000 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 01:37:06.063869 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:05.801024 2570 policy_none.go:47] "Start" Jul 22 01:37:06.063869 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.877769 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.063869 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:05.978089 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.184764 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.078436 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.184764 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.178964 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.301537 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.279969 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.423069 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.380567 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.546482 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.481338 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.668167 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.581945 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.686654 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.682534 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.796553 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:06.722862 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 01:37:06.796553 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.783326 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:06.796553 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:06.790569 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 01:32:06 +0000 UTC" deadline="2028-02-23 00:46:31.34908725 +0000 UTC" Jul 22 01:37:06.796553 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:06.790590 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13943h9m24.558500414s" Jul 22 01:37:06.933305 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.884233 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.060118 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:06.984777 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.089143 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.085778 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.197151 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.186382 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.315985 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.287414 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.429122 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.388330 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.488556 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.488483 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.488682 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.488654 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:07.490025 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.489556 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:07.491909 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.491886 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:07.493790 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.493724 2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-174.ec2.internal\" not found" node="ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.494621 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.494603 2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-174.ec2.internal" not found Jul 22 01:37:07.504051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.504031 2570 manager.go:337] "Starting Device Plugin manager" Jul 22 01:37:07.504146 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.504067 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 01:37:07.504146 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.504076 2570 server.go:87] "Starting device plugin registration server" Jul 22 01:37:07.504407 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.504391 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 01:37:07.504509 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.504407 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 01:37:07.504689 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.504486 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 01:37:07.504877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.504770 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 01:37:07.504877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.504796 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 01:37:07.505256 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.505237 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 01:37:07.505363 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.505293 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.510434 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.510414 2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-174.ec2.internal" not found Jul 22 01:37:07.571666 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.571641 2570 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-174.ec2.internal" not found Jul 22 01:37:07.592176 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.592151 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 01:37:07.592281 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.592192 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 01:37:07.592281 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.592213 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 01:37:07.592281 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.592244 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 01:37:07.594455 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.594431 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:07.605400 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.605385 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:07.606216 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.606198 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:07.606331 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.606251 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:07.606331 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.606268 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:07.606331 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.606292 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.617223 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.617199 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.617223 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.617220 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-174.ec2.internal\": node \"ip-10-0-130-174.ec2.internal\" not found" Jul 22 01:37:07.692918 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.692878 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal"] Jul 22 01:37:07.695601 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.695585 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.697811 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.697788 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.722031 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.722008 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.726529 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.726516 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.736766 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.736751 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 01:37:07.739611 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.739575 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 01:37:07.758128 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.758110 2570 apiserver.go:52] "Watching apiserver" Jul 22 01:37:07.762853 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.762839 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 01:37:07.765523 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.765492 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-wz9tm","openshift-ovn-kubernetes/ovnkube-node-jgthx","kube-system/konnectivity-agent-5nkpp","kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2rs85","openshift-dns/node-resolver-ms7nh","openshift-image-registry/node-ca-jvxs5","openshift-multus/multus-additional-cni-plugins-5lvdv","openshift-network-diagnostics/network-check-target-rpm7t","openshift-network-operator/iptables-alerter-jsflk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal"] Jul 22 01:37:07.768473 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.768455 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.771000 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.770979 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 01:37:07.771084 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.771035 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 01:37:07.771285 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.771270 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 01:37:07.771407 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.771388 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 01:37:07.771485 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.771469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.771537 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.771500 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s8r7s\"" Jul 22 01:37:07.773670 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.773651 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 01:37:07.773812 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.773797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:07.774125 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.774108 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 01:37:07.774203 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.774163 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 01:37:07.774278 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.774213 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 01:37:07.774348 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.774316 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 01:37:07.774744 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.774729 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f2jvm\"" Jul 22 01:37:07.774802 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.774768 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 01:37:07.776218 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.776203 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.776971 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.776954 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vhfhg\"" Jul 22 01:37:07.777042 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.776993 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 01:37:07.777080 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.776993 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 01:37:07.778053 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.778039 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 01:37:07.778724 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.778705 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.779195 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.779179 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 01:37:07.779250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.779216 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:37:07.779447 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.779435 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6hgjf\"" Jul 22 01:37:07.780741 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.780727 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f8bfk\"" Jul 22 01:37:07.780792 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.780732 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 01:37:07.780890 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.780876 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 01:37:07.781099 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.781083 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:07.783103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.783088 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 01:37:07.783167 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.783149 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 01:37:07.783204 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.783190 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 01:37:07.783265 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.783206 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mgwrh\"" Jul 22 01:37:07.784657 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.784639 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.786951 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.786933 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:07.787051 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.787026 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:07.787143 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.787128 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 01:37:07.787308 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.787275 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 01:37:07.787523 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.787510 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f9mlk\"" Jul 22 01:37:07.789334 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.789318 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-mx4nn"] Jul 22 01:37:07.789417 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.789404 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:07.791910 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.791894 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 01:37:07.791910 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.791901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e99a4a-3774-41c7-986d-c5b573d180a8-tmp\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.792075 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.791936 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-conf-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.792075 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.791949 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:37:07.792075 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792006 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-etc-kubernetes\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792108 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-run-netns\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-var-lib-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792221 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aa63b9d-84e0-4120-a6c3-75b7c6c1b929-konnectivity-ca\") pod \"konnectivity-agent-5nkpp\" (UID: \"1aa63b9d-84e0-4120-a6c3-75b7c6c1b929\") " pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792225 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-29tgn\"" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792247 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p"] Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-var-lib-kubelet\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792357 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-cnibin\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792437 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6sn\" (UniqueName: \"kubernetes.io/projected/24856620-eea2-46b7-9e0c-67fbe9604cd3-kube-api-access-sb6sn\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-ovn\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792538 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysctl-d\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792559 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-systemd\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9868bf39-940c-4e86-9405-1920fc525b8a-hosts-file\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.792635 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:07.792945 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-daemon-config\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.793737 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.792742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9868bf39-940c-4e86-9405-1920fc525b8a-tmp-dir\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.793737 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.793296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e7b3b1ca11b8cca62c69eddaf83e8b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal\" (UID: \"4e7b3b1ca11b8cca62c69eddaf83e8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.793737 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.793377 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-cni-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.793737 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.793443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-node-log\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.793737 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.793520 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-lib-modules\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.793737 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.793614 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e7b3b1ca11b8cca62c69eddaf83e8b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal\" (UID: \"4e7b3b1ca11b8cca62c69eddaf83e8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.795448 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795416 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-system-cni-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.795543 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795468 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-kubelet\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.795601 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795561 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkq2k\" (UniqueName: \"kubernetes.io/projected/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-kube-api-access-pkq2k\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.795601 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795570 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 01:37:07.795698 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovnkube-script-lib\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.795698 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795625 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-kubernetes\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.795698 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-socket-dir-parent\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.795898 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795709 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-slash\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.795898 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795852 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-etc-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.795898 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.796033 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795919 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-cni-bin\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.796033 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795939 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-log-socket\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.796033 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.795972 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-kubelet\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.796033 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796000 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-hostroot\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.796221 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-host\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.796221 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-systemd\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.796347 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796181 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-run-ovn-kubernetes\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.796347 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovnkube-config\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.796347 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796327 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-run\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.796501 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796377 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96c7n\" (UniqueName: \"kubernetes.io/projected/9868bf39-940c-4e86-9405-1920fc525b8a-kube-api-access-96c7n\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.796501 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-sys\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.796501 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-tuned\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.796641 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796541 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-cni-bin\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.796690 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-cni-multus\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.796740 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-multus-certs\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.796789 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796765 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.796839 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovn-node-metrics-cert\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.796891 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796835 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrf2\" (UniqueName: \"kubernetes.io/projected/02e99a4a-3774-41c7-986d-c5b573d180a8-kube-api-access-grrf2\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.796948 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.796909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysctl-conf\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.797043 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-k8s-cni-cncf-io\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.797084 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aa63b9d-84e0-4120-a6c3-75b7c6c1b929-agent-certs\") pod \"konnectivity-agent-5nkpp\" (UID: \"1aa63b9d-84e0-4120-a6c3-75b7c6c1b929\") " pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:07.797189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797164 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-netns\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.797574 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797552 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-modprobe-d\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.797655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797590 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/391551e060e624dcd7b1ada7a7b8652e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-174.ec2.internal\" (UID: \"391551e060e624dcd7b1ada7a7b8652e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.797655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24856620-eea2-46b7-9e0c-67fbe9604cd3-cni-binary-copy\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.797655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-cni-netd\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.797796 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797669 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysconfig\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.797796 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-os-release\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.797796 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-systemd-units\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.797796 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.797733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-env-overrides\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.798218 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.798203 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:07.800408 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.800387 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 01:37:07.800545 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.800526 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qmr5c\"" Jul 22 01:37:07.800735 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.800720 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 01:37:07.801020 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.801003 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 01:37:07.813921 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.813900 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7tb7b"] Jul 22 01:37:07.816549 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.816533 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:07.816619 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:07.816585 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:07.832121 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.832097 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nm9gh" Jul 22 01:37:07.843041 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.843019 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nm9gh" Jul 22 01:37:07.878853 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.878834 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 01:37:07.898498 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898476 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:07.898574 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrf2\" (UniqueName: \"kubernetes.io/projected/02e99a4a-3774-41c7-986d-c5b573d180a8-kube-api-access-grrf2\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.898574 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898554 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysctl-conf\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.898683 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898612 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-k8s-cni-cncf-io\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.898683 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898643 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aa63b9d-84e0-4120-a6c3-75b7c6c1b929-agent-certs\") pod \"konnectivity-agent-5nkpp\" (UID: \"1aa63b9d-84e0-4120-a6c3-75b7c6c1b929\") " pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:07.898683 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898663 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b220b990-79d7-46c0-aa4a-8d2ff554565c-host-slash\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:07.898829 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-netns\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.898829 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898695 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-k8s-cni-cncf-io\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.898829 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898700 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysctl-conf\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.898829 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898763 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-netns\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.898829 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-modprobe-d\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.898829 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898806 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/391551e060e624dcd7b1ada7a7b8652e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-174.ec2.internal\" (UID: \"391551e060e624dcd7b1ada7a7b8652e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24856620-eea2-46b7-9e0c-67fbe9604cd3-cni-binary-copy\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898862 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-cni-netd\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898854 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-modprobe-d\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898893 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898865 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/391551e060e624dcd7b1ada7a7b8652e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-174.ec2.internal\" (UID: \"391551e060e624dcd7b1ada7a7b8652e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898921 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898921 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-cni-netd\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45r75\" (UniqueName: \"kubernetes.io/projected/b220b990-79d7-46c0-aa4a-8d2ff554565c-kube-api-access-45r75\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.898988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysconfig\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899025 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-os-release\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899050 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-systemd-units\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.899096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-env-overrides\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899108 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-os-release\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899070 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysconfig\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899118 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-etc-selinux\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899158 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e99a4a-3774-41c7-986d-c5b573d180a8-tmp\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899161 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-systemd-units\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899182 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-conf-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899206 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-etc-kubernetes\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899293 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-conf-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899328 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-run-netns\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899346 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-etc-kubernetes\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-var-lib-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aa63b9d-84e0-4120-a6c3-75b7c6c1b929-konnectivity-ca\") pod \"konnectivity-agent-5nkpp\" (UID: \"1aa63b9d-84e0-4120-a6c3-75b7c6c1b929\") " pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899395 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-run-netns\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntl4\" (UniqueName: \"kubernetes.io/projected/541f80c8-3bbc-40d7-af2c-9d386d3b946c-kube-api-access-5ntl4\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:07.899575 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899441 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-var-lib-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlbg\" (UniqueName: \"kubernetes.io/projected/5f9f1919-de93-4f46-a439-055b29894833-kube-api-access-qwlbg\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899491 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-var-lib-kubelet\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24856620-eea2-46b7-9e0c-67fbe9604cd3-cni-binary-copy\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-cnibin\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899581 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899611 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-env-overrides\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6sn\" (UniqueName: \"kubernetes.io/projected/24856620-eea2-46b7-9e0c-67fbe9604cd3-kube-api-access-sb6sn\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-ovn\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899616 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-cnibin\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysctl-d\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-systemd\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899738 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-ovn\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9868bf39-940c-4e86-9405-1920fc525b8a-hosts-file\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899609 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-var-lib-kubelet\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.900328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899798 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-daemon-config\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9868bf39-940c-4e86-9405-1920fc525b8a-tmp-dir\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899884 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-sysctl-d\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899873 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9868bf39-940c-4e86-9405-1920fc525b8a-hosts-file\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899787 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-systemd\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.899963 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e7b3b1ca11b8cca62c69eddaf83e8b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal\" (UID: \"4e7b3b1ca11b8cca62c69eddaf83e8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-cni-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-node-log\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900036 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e7b3b1ca11b8cca62c69eddaf83e8b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal\" (UID: \"4e7b3b1ca11b8cca62c69eddaf83e8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900055 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-os-release\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900140 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-lib-modules\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900148 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9868bf39-940c-4e86-9405-1920fc525b8a-tmp-dir\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900162 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-cni-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.901103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900165 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e7b3b1ca11b8cca62c69eddaf83e8b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal\" (UID: \"4e7b3b1ca11b8cca62c69eddaf83e8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900205 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-node-log\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-system-cni-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900226 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e7b3b1ca11b8cca62c69eddaf83e8b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal\" (UID: \"4e7b3b1ca11b8cca62c69eddaf83e8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900275 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-kubelet\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkq2k\" (UniqueName: \"kubernetes.io/projected/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-kube-api-access-pkq2k\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900327 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-daemon-config\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900327 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-system-cni-dir\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900321 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-lib-modules\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4v6t\" (UniqueName: \"kubernetes.io/projected/094e4ffe-4ef6-49e6-baf9-8459b8e67813-kube-api-access-r4v6t\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900378 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-kubelet\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovnkube-script-lib\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-kubernetes\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900593 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-socket-dir-parent\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.901778 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-slash\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-etc-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900663 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-multus-socket-dir-parent\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900697 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-slash\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900727 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-kubernetes\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-cni-bin\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900791 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-etc-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900789 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cnibin\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aa63b9d-84e0-4120-a6c3-75b7c6c1b929-konnectivity-ca\") pod \"konnectivity-agent-5nkpp\" (UID: \"1aa63b9d-84e0-4120-a6c3-75b7c6c1b929\") " pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900841 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-cni-bin\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-log-socket\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900869 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-log-socket\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900878 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-openvswitch\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.902487 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900918 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b220b990-79d7-46c0-aa4a-8d2ff554565c-iptables-alerter-script\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900949 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nqq\" (UniqueName: \"kubernetes.io/projected/aebe006d-9a1d-4049-b3e1-37130f7339aa-kube-api-access-d4nqq\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900975 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.900999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-sys-fs\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901026 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-kubelet\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901051 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-hostroot\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901059 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovnkube-script-lib\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901076 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aebe006d-9a1d-4049-b3e1-37130f7339aa-host\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901112 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-hostroot\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901112 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-kubelet\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901134 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901169 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-host\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-systemd\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901252 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-run-systemd\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-host\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.903170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901330 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-run-ovn-kubernetes\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovnkube-config\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901397 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-run-ovn-kubernetes\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aebe006d-9a1d-4049-b3e1-37130f7339aa-serviceca\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-socket-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901485 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-registration-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901515 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-run\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901561 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-run\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96c7n\" (UniqueName: \"kubernetes.io/projected/9868bf39-940c-4e86-9405-1920fc525b8a-kube-api-access-96c7n\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901640 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21b6ba48-2cfc-4b03-a77d-0ced99379d04-kubelet-config\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21b6ba48-2cfc-4b03-a77d-0ced99379d04-dbus\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-device-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901862 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-sys\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.904051 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-tuned\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901937 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovnkube-config\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901943 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901938 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e99a4a-3774-41c7-986d-c5b573d180a8-sys\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.901981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-cni-bin\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-cni-multus\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902047 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-multus-certs\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902099 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-cni-bin\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovn-node-metrics-cert\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902144 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-system-cni-dir\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902148 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-var-lib-cni-multus\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902253 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24856620-eea2-46b7-9e0c-67fbe9604cd3-host-run-multus-certs\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.902296 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.904877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.903477 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e99a4a-3774-41c7-986d-c5b573d180a8-tmp\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.905610 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.903820 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aa63b9d-84e0-4120-a6c3-75b7c6c1b929-agent-certs\") pod \"konnectivity-agent-5nkpp\" (UID: \"1aa63b9d-84e0-4120-a6c3-75b7c6c1b929\") " pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:07.905610 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.904331 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02e99a4a-3774-41c7-986d-c5b573d180a8-etc-tuned\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.905610 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.904989 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-ovn-node-metrics-cert\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.912939 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.912915 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrf2\" (UniqueName: \"kubernetes.io/projected/02e99a4a-3774-41c7-986d-c5b573d180a8-kube-api-access-grrf2\") pod \"tuned-2rs85\" (UID: \"02e99a4a-3774-41c7-986d-c5b573d180a8\") " pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:07.914362 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.914340 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96c7n\" (UniqueName: \"kubernetes.io/projected/9868bf39-940c-4e86-9405-1920fc525b8a-kube-api-access-96c7n\") pod \"node-resolver-ms7nh\" (UID: \"9868bf39-940c-4e86-9405-1920fc525b8a\") " pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:07.915428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.915406 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkq2k\" (UniqueName: \"kubernetes.io/projected/0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082-kube-api-access-pkq2k\") pod \"ovnkube-node-jgthx\" (UID: \"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082\") " pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:07.915608 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:07.915593 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6sn\" (UniqueName: \"kubernetes.io/projected/24856620-eea2-46b7-9e0c-67fbe9604cd3-kube-api-access-sb6sn\") pod \"multus-wz9tm\" (UID: \"24856620-eea2-46b7-9e0c-67fbe9604cd3\") " pod="openshift-multus/multus-wz9tm" Jul 22 01:37:08.003017 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.002943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-os-release\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.003017 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.002973 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:08.003017 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.002992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4v6t\" (UniqueName: \"kubernetes.io/projected/094e4ffe-4ef6-49e6-baf9-8459b8e67813-kube-api-access-r4v6t\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.003017 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003012 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cnibin\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b220b990-79d7-46c0-aa4a-8d2ff554565c-iptables-alerter-script\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003051 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nqq\" (UniqueName: \"kubernetes.io/projected/aebe006d-9a1d-4049-b3e1-37130f7339aa-kube-api-access-d4nqq\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003071 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-os-release\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003096 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cnibin\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-sys-fs\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aebe006d-9a1d-4049-b3e1-37130f7339aa-host\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.003116 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003195 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aebe006d-9a1d-4049-b3e1-37130f7339aa-serviceca\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-socket-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003250 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aebe006d-9a1d-4049-b3e1-37130f7339aa-host\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-registration-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.003607 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.003315 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret podName:21b6ba48-2cfc-4b03-a77d-0ced99379d04 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:08.503255762 +0000 UTC m=+3.000697111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret") pod "global-pull-secret-syncer-7tb7b" (UID: "21b6ba48-2cfc-4b03-a77d-0ced99379d04") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003319 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-registration-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.003181 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.003372 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs podName:541f80c8-3bbc-40d7-af2c-9d386d3b946c nodeName:}" failed. No retries permitted until 2026-07-22 01:37:08.503359653 +0000 UTC m=+3.000800990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs") pod "network-metrics-daemon-mx4nn" (UID: "541f80c8-3bbc-40d7-af2c-9d386d3b946c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003396 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-sys-fs\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003412 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-socket-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21b6ba48-2cfc-4b03-a77d-0ced99379d04-kubelet-config\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003465 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21b6ba48-2cfc-4b03-a77d-0ced99379d04-dbus\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003467 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21b6ba48-2cfc-4b03-a77d-0ced99379d04-kubelet-config\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-device-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003537 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-device-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-system-cni-dir\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.004210 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003593 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21b6ba48-2cfc-4b03-a77d-0ced99379d04-dbus\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003637 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-system-cni-dir\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b220b990-79d7-46c0-aa4a-8d2ff554565c-host-slash\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003670 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b220b990-79d7-46c0-aa4a-8d2ff554565c-host-slash\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003694 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094e4ffe-4ef6-49e6-baf9-8459b8e67813-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003717 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003731 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aebe006d-9a1d-4049-b3e1-37130f7339aa-serviceca\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45r75\" (UniqueName: \"kubernetes.io/projected/b220b990-79d7-46c0-aa4a-8d2ff554565c-kube-api-access-45r75\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-etc-selinux\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003801 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b220b990-79d7-46c0-aa4a-8d2ff554565c-iptables-alerter-script\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntl4\" (UniqueName: \"kubernetes.io/projected/541f80c8-3bbc-40d7-af2c-9d386d3b946c-kube-api-access-5ntl4\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:08.004655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlbg\" (UniqueName: \"kubernetes.io/projected/5f9f1919-de93-4f46-a439-055b29894833-kube-api-access-qwlbg\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.005068 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003882 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.005068 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.003940 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f9f1919-de93-4f46-a439-055b29894833-etc-selinux\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.005068 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.004192 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.005068 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.004191 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/094e4ffe-4ef6-49e6-baf9-8459b8e67813-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.013756 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.013731 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4v6t\" (UniqueName: \"kubernetes.io/projected/094e4ffe-4ef6-49e6-baf9-8459b8e67813-kube-api-access-r4v6t\") pod \"multus-additional-cni-plugins-5lvdv\" (UID: \"094e4ffe-4ef6-49e6-baf9-8459b8e67813\") " pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.017390 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.017371 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:08.017390 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.017391 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:08.017632 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.017402 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2m4rn for pod openshift-network-diagnostics/network-check-target-rpm7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:08.017632 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.017481 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn podName:f636171e-37d9-4c74-bd11-512f7ac26153 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:08.517469139 +0000 UTC m=+3.014910480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2m4rn" (UniqueName: "kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn") pod "network-check-target-rpm7t" (UID: "f636171e-37d9-4c74-bd11-512f7ac26153") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:08.019791 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.019768 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlbg\" (UniqueName: \"kubernetes.io/projected/5f9f1919-de93-4f46-a439-055b29894833-kube-api-access-qwlbg\") pod \"aws-ebs-csi-driver-node-ph24p\" (UID: \"5f9f1919-de93-4f46-a439-055b29894833\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.019965 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.019943 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45r75\" (UniqueName: \"kubernetes.io/projected/b220b990-79d7-46c0-aa4a-8d2ff554565c-kube-api-access-45r75\") pod \"iptables-alerter-jsflk\" (UID: \"b220b990-79d7-46c0-aa4a-8d2ff554565c\") " pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:08.020687 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.020666 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntl4\" (UniqueName: \"kubernetes.io/projected/541f80c8-3bbc-40d7-af2c-9d386d3b946c-kube-api-access-5ntl4\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:08.020909 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.020891 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nqq\" (UniqueName: \"kubernetes.io/projected/aebe006d-9a1d-4049-b3e1-37130f7339aa-kube-api-access-d4nqq\") pod \"node-ca-jvxs5\" (UID: \"aebe006d-9a1d-4049-b3e1-37130f7339aa\") " pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:08.040148 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.040120 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" Jul 22 01:37:08.043369 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.043351 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" Jul 22 01:37:08.102111 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.102075 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wz9tm" Jul 22 01:37:08.108137 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.108119 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:08.133800 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.133773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:08.138642 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.138621 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2rs85" Jul 22 01:37:08.165584 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.165551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ms7nh" Jul 22 01:37:08.185141 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.185115 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvxs5" Jul 22 01:37:08.200618 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.200592 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" Jul 22 01:37:08.231853 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.231820 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jsflk" Jul 22 01:37:08.238938 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.238917 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" Jul 22 01:37:08.509466 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.509442 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:08.509618 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.509486 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:08.509618 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.509601 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:08.509687 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.509653 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs podName:541f80c8-3bbc-40d7-af2c-9d386d3b946c nodeName:}" failed. No retries permitted until 2026-07-22 01:37:09.509640542 +0000 UTC m=+4.007081878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs") pod "network-metrics-daemon-mx4nn" (UID: "541f80c8-3bbc-40d7-af2c-9d386d3b946c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:08.509732 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.509603 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:08.509772 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.509761 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret podName:21b6ba48-2cfc-4b03-a77d-0ced99379d04 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:09.50973482 +0000 UTC m=+4.007176166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret") pod "global-pull-secret-syncer-7tb7b" (UID: "21b6ba48-2cfc-4b03-a77d-0ced99379d04") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:08.610820 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.610791 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:08.611058 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.610915 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:08.611058 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.610930 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:08.611058 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.610940 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2m4rn for pod openshift-network-diagnostics/network-check-target-rpm7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:08.611058 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:08.610992 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn podName:f636171e-37d9-4c74-bd11-512f7ac26153 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:09.610978309 +0000 UTC m=+4.108419646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m4rn" (UniqueName: "kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn") pod "network-check-target-rpm7t" (UID: "f636171e-37d9-4c74-bd11-512f7ac26153") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:08.774331 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:08.774287 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa63b9d_84e0_4120_a6c3_75b7c6c1b929.slice/crio-37fee6f69d07ac4159770e10dd2ecd0063fec57a9456bac7a5392dc45454b06f WatchSource:0}: Error finding container 37fee6f69d07ac4159770e10dd2ecd0063fec57a9456bac7a5392dc45454b06f: Status 404 returned error can't find the container with id 37fee6f69d07ac4159770e10dd2ecd0063fec57a9456bac7a5392dc45454b06f Jul 22 01:37:08.774601 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:08.774580 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7b3b1ca11b8cca62c69eddaf83e8b0.slice/crio-f8116ba8438f245b97b5d9c8721ee6701bab113c49dc283bf1f110a7620d87d3 WatchSource:0}: Error finding container f8116ba8438f245b97b5d9c8721ee6701bab113c49dc283bf1f110a7620d87d3: Status 404 returned error can't find the container with id f8116ba8438f245b97b5d9c8721ee6701bab113c49dc283bf1f110a7620d87d3 Jul 22 01:37:08.780694 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.780672 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:37:08.783748 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:08.783677 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb220b990_79d7_46c0_aa4a_8d2ff554565c.slice/crio-d991fd9cbdc0edd03e0a0a759fce910853d8a0c6ea4ad5fa07a378bd2af83c20 WatchSource:0}: Error finding container d991fd9cbdc0edd03e0a0a759fce910853d8a0c6ea4ad5fa07a378bd2af83c20: Status 404 returned error can't find the container with id d991fd9cbdc0edd03e0a0a759fce910853d8a0c6ea4ad5fa07a378bd2af83c20 Jul 22 01:37:08.784808 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:08.784788 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebe006d_9a1d_4049_b3e1_37130f7339aa.slice/crio-e2ac5defe6f232824911a0e7658893a2b06d105da575f3595a8044d21e075cbd WatchSource:0}: Error finding container e2ac5defe6f232824911a0e7658893a2b06d105da575f3595a8044d21e075cbd: Status 404 returned error can't find the container with id e2ac5defe6f232824911a0e7658893a2b06d105da575f3595a8044d21e075cbd Jul 22 01:37:08.790799 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:08.790651 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094e4ffe_4ef6_49e6_baf9_8459b8e67813.slice/crio-31fd61f104ef535ea2c8e2280659a2598dab93e152920d7af0345d3d7d67f66f WatchSource:0}: Error finding container 31fd61f104ef535ea2c8e2280659a2598dab93e152920d7af0345d3d7d67f66f: Status 404 returned error can't find the container with id 31fd61f104ef535ea2c8e2280659a2598dab93e152920d7af0345d3d7d67f66f Jul 22 01:37:08.791770 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:08.791736 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24856620_eea2_46b7_9e0c_67fbe9604cd3.slice/crio-d3128db8f522d0d6e5f11d3391c5f231ca3c81dc857d8f668657863c369ae5f4 WatchSource:0}: Error finding container d3128db8f522d0d6e5f11d3391c5f231ca3c81dc857d8f668657863c369ae5f4: Status 404 returned error can't find the container with id d3128db8f522d0d6e5f11d3391c5f231ca3c81dc857d8f668657863c369ae5f4 Jul 22 01:37:08.847105 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.847068 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:32:08 +0000 UTC" deadline="2028-02-19 09:15:23.923308684 +0000 UTC" Jul 22 01:37:08.847105 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:08.847099 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13855h38m15.076215271s" Jul 22 01:37:08.877373 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:08.877351 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c717bdb_d2b6_4bf6_84fd_1b0e9d0ba082.slice/crio-1f2eb3df9b3af2891f93eb9819319c92c14dd44fcd06f73d0b890c17be614151 WatchSource:0}: Error finding container 1f2eb3df9b3af2891f93eb9819319c92c14dd44fcd06f73d0b890c17be614151: Status 404 returned error can't find the container with id 1f2eb3df9b3af2891f93eb9819319c92c14dd44fcd06f73d0b890c17be614151 Jul 22 01:37:08.956547 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:08.956502 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391551e060e624dcd7b1ada7a7b8652e.slice/crio-c39dccda01f708f893fd479414986735d493a33bddd2228c3b35de8360181b87 WatchSource:0}: Error finding container c39dccda01f708f893fd479414986735d493a33bddd2228c3b35de8360181b87: Status 404 returned error can't find the container with id c39dccda01f708f893fd479414986735d493a33bddd2228c3b35de8360181b87 Jul 22 01:37:09.195853 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:09.195821 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9868bf39_940c_4e86_9405_1920fc525b8a.slice/crio-6b254e8d73d399bf04015f6071c4fd3018a7cb6bf96a33ade05c8fc67fe49c3b WatchSource:0}: Error finding container 6b254e8d73d399bf04015f6071c4fd3018a7cb6bf96a33ade05c8fc67fe49c3b: Status 404 returned error can't find the container with id 6b254e8d73d399bf04015f6071c4fd3018a7cb6bf96a33ade05c8fc67fe49c3b Jul 22 01:37:09.216253 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:09.216204 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e99a4a_3774_41c7_986d_c5b573d180a8.slice/crio-12c93f29c8eda3b21ed07ae67cdd01b1e6c98c4d609a8a62475a8401f72691d3 WatchSource:0}: Error finding container 12c93f29c8eda3b21ed07ae67cdd01b1e6c98c4d609a8a62475a8401f72691d3: Status 404 returned error can't find the container with id 12c93f29c8eda3b21ed07ae67cdd01b1e6c98c4d609a8a62475a8401f72691d3 Jul 22 01:37:09.264089 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:09.264054 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9f1919_de93_4f46_a439_055b29894833.slice/crio-8f67b7b284329218a97e900f6ee999e6fac6e5298a0dbd0e0786e47efa07962e WatchSource:0}: Error finding container 8f67b7b284329218a97e900f6ee999e6fac6e5298a0dbd0e0786e47efa07962e: Status 404 returned error can't find the container with id 8f67b7b284329218a97e900f6ee999e6fac6e5298a0dbd0e0786e47efa07962e Jul 22 01:37:09.518704 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.518613 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:09.518704 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.518680 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:09.518939 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.518852 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:09.518939 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.518920 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs podName:541f80c8-3bbc-40d7-af2c-9d386d3b946c nodeName:}" failed. No retries permitted until 2026-07-22 01:37:11.518899252 +0000 UTC m=+6.016340592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs") pod "network-metrics-daemon-mx4nn" (UID: "541f80c8-3bbc-40d7-af2c-9d386d3b946c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:09.519396 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.519287 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:09.519396 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.519357 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret podName:21b6ba48-2cfc-4b03-a77d-0ced99379d04 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:11.519338726 +0000 UTC m=+6.016780068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret") pod "global-pull-secret-syncer-7tb7b" (UID: "21b6ba48-2cfc-4b03-a77d-0ced99379d04") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:09.593393 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.593363 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:09.593579 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.593494 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:09.594003 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.593981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:09.594126 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.594106 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:09.596573 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.596545 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:09.596701 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.596647 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:09.602157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.602107 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"1f2eb3df9b3af2891f93eb9819319c92c14dd44fcd06f73d0b890c17be614151"} Jul 22 01:37:09.604949 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.604923 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wz9tm" event={"ID":"24856620-eea2-46b7-9e0c-67fbe9604cd3","Type":"ContainerStarted","Data":"d3128db8f522d0d6e5f11d3391c5f231ca3c81dc857d8f668657863c369ae5f4"} Jul 22 01:37:09.608484 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.608456 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" event={"ID":"094e4ffe-4ef6-49e6-baf9-8459b8e67813","Type":"ContainerStarted","Data":"31fd61f104ef535ea2c8e2280659a2598dab93e152920d7af0345d3d7d67f66f"} Jul 22 01:37:09.614422 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.614311 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jsflk" event={"ID":"b220b990-79d7-46c0-aa4a-8d2ff554565c","Type":"ContainerStarted","Data":"d991fd9cbdc0edd03e0a0a759fce910853d8a0c6ea4ad5fa07a378bd2af83c20"} Jul 22 01:37:09.617047 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.616968 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5nkpp" event={"ID":"1aa63b9d-84e0-4120-a6c3-75b7c6c1b929","Type":"ContainerStarted","Data":"37fee6f69d07ac4159770e10dd2ecd0063fec57a9456bac7a5392dc45454b06f"} Jul 22 01:37:09.620221 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.619765 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:09.620221 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.619964 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:09.620221 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.619986 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:09.620221 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.620000 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2m4rn for pod openshift-network-diagnostics/network-check-target-rpm7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:09.620221 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:09.620057 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn podName:f636171e-37d9-4c74-bd11-512f7ac26153 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:11.620038446 +0000 UTC m=+6.117479799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m4rn" (UniqueName: "kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn") pod "network-check-target-rpm7t" (UID: "f636171e-37d9-4c74-bd11-512f7ac26153") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:09.624535 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.624445 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2rs85" event={"ID":"02e99a4a-3774-41c7-986d-c5b573d180a8","Type":"ContainerStarted","Data":"12c93f29c8eda3b21ed07ae67cdd01b1e6c98c4d609a8a62475a8401f72691d3"} Jul 22 01:37:09.627030 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.626991 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" event={"ID":"391551e060e624dcd7b1ada7a7b8652e","Type":"ContainerStarted","Data":"c39dccda01f708f893fd479414986735d493a33bddd2228c3b35de8360181b87"} Jul 22 01:37:09.632534 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.632507 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvxs5" event={"ID":"aebe006d-9a1d-4049-b3e1-37130f7339aa","Type":"ContainerStarted","Data":"e2ac5defe6f232824911a0e7658893a2b06d105da575f3595a8044d21e075cbd"} Jul 22 01:37:09.635165 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.635141 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" event={"ID":"4e7b3b1ca11b8cca62c69eddaf83e8b0","Type":"ContainerStarted","Data":"f8116ba8438f245b97b5d9c8721ee6701bab113c49dc283bf1f110a7620d87d3"} Jul 22 01:37:09.639526 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.639501 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" event={"ID":"5f9f1919-de93-4f46-a439-055b29894833","Type":"ContainerStarted","Data":"8f67b7b284329218a97e900f6ee999e6fac6e5298a0dbd0e0786e47efa07962e"} Jul 22 01:37:09.645591 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.645564 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ms7nh" event={"ID":"9868bf39-940c-4e86-9405-1920fc525b8a","Type":"ContainerStarted","Data":"6b254e8d73d399bf04015f6071c4fd3018a7cb6bf96a33ade05c8fc67fe49c3b"} Jul 22 01:37:09.848032 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.847914 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:32:08 +0000 UTC" deadline="2028-01-20 07:00:02.554670347 +0000 UTC" Jul 22 01:37:09.848032 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:09.847982 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13133h22m52.706693053s" Jul 22 01:37:11.536151 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:11.535852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:11.536151 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:11.535956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:11.536151 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.536031 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:11.536151 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.536068 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:11.536151 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.536109 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs podName:541f80c8-3bbc-40d7-af2c-9d386d3b946c nodeName:}" failed. No retries permitted until 2026-07-22 01:37:15.53608883 +0000 UTC m=+10.033530172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs") pod "network-metrics-daemon-mx4nn" (UID: "541f80c8-3bbc-40d7-af2c-9d386d3b946c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:11.536151 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.536130 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret podName:21b6ba48-2cfc-4b03-a77d-0ced99379d04 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:15.536119684 +0000 UTC m=+10.033561026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret") pod "global-pull-secret-syncer-7tb7b" (UID: "21b6ba48-2cfc-4b03-a77d-0ced99379d04") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:11.596393 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:11.593973 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:11.596393 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.594136 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:11.596393 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:11.594582 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:11.596393 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.594682 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:11.596393 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:11.595047 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:11.596393 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.595152 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:11.637209 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:11.636577 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:11.637209 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.636753 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:11.637209 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.636778 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:11.637209 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.636794 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2m4rn for pod openshift-network-diagnostics/network-check-target-rpm7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:11.637209 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:11.636854 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn podName:f636171e-37d9-4c74-bd11-512f7ac26153 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:15.636834173 +0000 UTC m=+10.134275515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m4rn" (UniqueName: "kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn") pod "network-check-target-rpm7t" (UID: "f636171e-37d9-4c74-bd11-512f7ac26153") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:13.592776 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:13.592743 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:13.593307 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:13.592887 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:13.593395 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:13.593299 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:13.593455 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:13.593415 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:13.593518 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:13.593495 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:13.593658 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:13.593628 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:15.578396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:15.577615 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:15.578396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:15.577675 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:15.578396 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.577827 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:15.578396 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.577891 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs podName:541f80c8-3bbc-40d7-af2c-9d386d3b946c nodeName:}" failed. No retries permitted until 2026-07-22 01:37:23.577869271 +0000 UTC m=+18.075310618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs") pod "network-metrics-daemon-mx4nn" (UID: "541f80c8-3bbc-40d7-af2c-9d386d3b946c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:15.578396 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.578298 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:15.578396 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.578355 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret podName:21b6ba48-2cfc-4b03-a77d-0ced99379d04 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:23.578338718 +0000 UTC m=+18.075780073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret") pod "global-pull-secret-syncer-7tb7b" (UID: "21b6ba48-2cfc-4b03-a77d-0ced99379d04") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:15.593244 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:15.593043 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:15.593244 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:15.593148 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:15.593244 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.593161 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:15.593244 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:15.593044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:15.593568 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.593261 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:15.593568 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.593350 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:15.678901 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:15.678843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:15.679097 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.679068 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:15.679097 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.679098 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:15.679305 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.679113 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2m4rn for pod openshift-network-diagnostics/network-check-target-rpm7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:15.679305 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:15.679176 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn podName:f636171e-37d9-4c74-bd11-512f7ac26153 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:23.679159408 +0000 UTC m=+18.176600746 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m4rn" (UniqueName: "kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn") pod "network-check-target-rpm7t" (UID: "f636171e-37d9-4c74-bd11-512f7ac26153") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:17.594194 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:17.594162 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:17.594598 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:17.594306 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:17.594598 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:17.594378 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:17.594598 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:17.594423 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:17.594598 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:17.594497 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:17.594598 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:17.594568 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:19.593086 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:19.593053 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:19.593480 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:19.593053 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:19.593480 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:19.593183 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:19.593480 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:19.593234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:19.593480 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:19.593331 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:19.593480 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:19.593408 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:21.593013 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:21.592964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:21.593013 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:21.592997 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:21.593546 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:21.592964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:21.593546 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:21.593099 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:21.593546 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:21.593168 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:21.593546 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:21.593236 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:23.593442 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:23.593248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:23.593890 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:23.593248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:23.593890 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.593537 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:23.593890 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.593624 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:23.593890 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:23.593248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:23.593890 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.593722 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:23.641186 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:23.641141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:23.641368 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:23.641201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:23.641368 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.641323 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:23.641368 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.641352 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:23.641522 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.641402 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret podName:21b6ba48-2cfc-4b03-a77d-0ced99379d04 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:39.64137858 +0000 UTC m=+34.138819958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret") pod "global-pull-secret-syncer-7tb7b" (UID: "21b6ba48-2cfc-4b03-a77d-0ced99379d04") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:23.641522 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.641422 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs podName:541f80c8-3bbc-40d7-af2c-9d386d3b946c nodeName:}" failed. No retries permitted until 2026-07-22 01:37:39.641412557 +0000 UTC m=+34.138853892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs") pod "network-metrics-daemon-mx4nn" (UID: "541f80c8-3bbc-40d7-af2c-9d386d3b946c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:23.742069 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:23.742035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:23.742258 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.742232 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:23.742258 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.742258 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:23.742349 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.742270 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2m4rn for pod openshift-network-diagnostics/network-check-target-rpm7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:23.742349 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:23.742321 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn podName:f636171e-37d9-4c74-bd11-512f7ac26153 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:39.742308567 +0000 UTC m=+34.239749902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m4rn" (UniqueName: "kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn") pod "network-check-target-rpm7t" (UID: "f636171e-37d9-4c74-bd11-512f7ac26153") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:25.592777 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:25.592716 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:25.592777 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:25.592759 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:25.593286 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:25.592729 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:25.593286 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:25.592842 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:25.593286 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:25.592936 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:25.593286 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:25.593010 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:27.594014 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:27.593979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:27.594478 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:27.594064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:27.594478 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:27.594076 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:27.594478 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:27.594193 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:27.594478 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:27.594248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:27.594478 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:27.594342 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:29.593260 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:29.593233 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:29.593695 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:29.593269 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:29.593695 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:29.593343 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:29.593819 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:29.593797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:29.593887 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:29.593825 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:29.593948 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:29.593928 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:29.713962 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:29.713925 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2rs85" event={"ID":"02e99a4a-3774-41c7-986d-c5b573d180a8","Type":"ContainerStarted","Data":"df0c57fc4743124a8dd7283f31b03f10dded61c866d0c1c789fb9efe5400e9e3"} Jul 22 01:37:29.719803 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:29.719769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" event={"ID":"391551e060e624dcd7b1ada7a7b8652e","Type":"ContainerStarted","Data":"2f61f9bbc3c457c25316298d2cbc7fa785dde1ebe8dd7876063a801409d9cd26"} Jul 22 01:37:29.729399 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:29.729366 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wz9tm" event={"ID":"24856620-eea2-46b7-9e0c-67fbe9604cd3","Type":"ContainerStarted","Data":"6b3a4c160ac519d72f3f909bc99980c12ca271a34fc7da3500dfa63147e50bf4"} Jul 22 01:37:29.733874 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:29.733819 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2rs85" podStartSLOduration=1.5208215950000001 podStartE2EDuration="21.733800209s" totalImagesPullingTime="20.212978614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:29.733089492 +0000 UTC m=+24.230530850" watchObservedRunningTime="2026-07-22 01:37:29.733800209 +0000 UTC m=+24.231241566" Jul 22 01:37:29.750208 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:29.750165 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wz9tm" podStartSLOduration=1.111620804 podStartE2EDuration="21.750151046s" totalImagesPullingTime="20.638530242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:29.749977896 +0000 UTC m=+24.247419252" watchObservedRunningTime="2026-07-22 01:37:29.750151046 +0000 UTC m=+24.247592402" Jul 22 01:37:30.733465 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.733162 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvxs5" event={"ID":"aebe006d-9a1d-4049-b3e1-37130f7339aa","Type":"ContainerStarted","Data":"5902aef60351e6c8698891d35dfa41efbbac6116b0587f69d44f0f10a420f63e"} Jul 22 01:37:30.734697 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.734675 2570 generic.go:356] "Generic (PLEG): container finished" podID="4e7b3b1ca11b8cca62c69eddaf83e8b0" containerID="3c9d82d34e681ecb32f16d813dfcf26800674361b4441a669e0e7142d1a3ed1f" exitCode=0 Jul 22 01:37:30.734799 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.734749 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" event={"ID":"4e7b3b1ca11b8cca62c69eddaf83e8b0","Type":"ContainerDied","Data":"3c9d82d34e681ecb32f16d813dfcf26800674361b4441a669e0e7142d1a3ed1f"} Jul 22 01:37:30.736087 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.736060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" event={"ID":"5f9f1919-de93-4f46-a439-055b29894833","Type":"ContainerStarted","Data":"38c0a336159cc3156388b96f6800c73aabf6bd1f2101a714f60fc4fb3c8d6fdd"} Jul 22 01:37:30.737560 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.737542 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ms7nh" event={"ID":"9868bf39-940c-4e86-9405-1920fc525b8a","Type":"ContainerStarted","Data":"24ee3a19918ca54c27b29408d90919d17c79c13d21ecf9d45d0291cac3960048"} Jul 22 01:37:30.743041 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.743022 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:37:30.743367 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.743349 2570 generic.go:356] "Generic (PLEG): container finished" podID="0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082" containerID="0d153937bfba5fc889f814c15758ec56c500417d2848326d6e5d7099f04db373" exitCode=1 Jul 22 01:37:30.743432 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.743408 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"644d7e21f533086e9f3c20d4d549252ca9f043371d2ef0e44fade2112234d360"} Jul 22 01:37:30.743497 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.743485 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"2f65cb4f3add2a972edd568d28769df1d64c673f14c1c85ca24373ad2598de0c"} Jul 22 01:37:30.743533 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.743518 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"53b311cfefbf4501341081bc97357b8a15b1688fbfe4a305065eac67271569e7"} Jul 22 01:37:30.743572 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.743551 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"7c1227e61e502674c251fa774b29148f7c6ef9969d9bb99c8f11a50ca918d144"} Jul 22 01:37:30.743618 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.743577 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerDied","Data":"0d153937bfba5fc889f814c15758ec56c500417d2848326d6e5d7099f04db373"} Jul 22 01:37:30.743618 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.743613 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"fe99064df3d58d13164dd5abf3e1ba42a17112edc9a70617a9139872dbb1d04f"} Jul 22 01:37:30.744750 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.744729 2570 generic.go:356] "Generic (PLEG): container finished" podID="094e4ffe-4ef6-49e6-baf9-8459b8e67813" containerID="4fd5a6c742140aa5ea8f41d32dd0006e24271d587b650dad208f37bb64c2e23e" exitCode=0 Jul 22 01:37:30.744845 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.744790 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" event={"ID":"094e4ffe-4ef6-49e6-baf9-8459b8e67813","Type":"ContainerDied","Data":"4fd5a6c742140aa5ea8f41d32dd0006e24271d587b650dad208f37bb64c2e23e"} Jul 22 01:37:30.746245 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.746227 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jsflk" event={"ID":"b220b990-79d7-46c0-aa4a-8d2ff554565c","Type":"ContainerStarted","Data":"6eb46ec0ce687f725a7c73841956500b303cbe6dfe5ed59b6c9b309d39e9ccc1"} Jul 22 01:37:30.747701 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.747655 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5nkpp" event={"ID":"1aa63b9d-84e0-4120-a6c3-75b7c6c1b929","Type":"ContainerStarted","Data":"741c204ed45818733b04022521147a3bde377ccb8c2eaa524679f8eb174052a2"} Jul 22 01:37:30.748528 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.748486 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-174.ec2.internal" podStartSLOduration=22.748475544 podStartE2EDuration="22.748475544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:29.763255987 +0000 UTC m=+24.260697370" watchObservedRunningTime="2026-07-22 01:37:30.748475544 +0000 UTC m=+25.245916916" Jul 22 01:37:30.748647 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.748629 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jvxs5" podStartSLOduration=2.121766623 podStartE2EDuration="22.748624569s" totalImagesPullingTime="20.626857946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:30.74797585 +0000 UTC m=+25.245417233" watchObservedRunningTime="2026-07-22 01:37:30.748624569 +0000 UTC m=+25.246065925" Jul 22 01:37:30.781791 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.781734 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jsflk" podStartSLOduration=2.170781519 podStartE2EDuration="22.781713269s" totalImagesPullingTime="20.61093175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:30.781551521 +0000 UTC m=+25.278992881" watchObservedRunningTime="2026-07-22 01:37:30.781713269 +0000 UTC m=+25.279154626" Jul 22 01:37:30.797583 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:30.797537 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5nkpp" podStartSLOduration=6.85360428 podStartE2EDuration="22.797512338s" totalImagesPullingTime="15.943908058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:30.797428415 +0000 UTC m=+25.294869772" watchObservedRunningTime="2026-07-22 01:37:30.797512338 +0000 UTC m=+25.294953732" Jul 22 01:37:31.284884 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.284844 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:31.371950 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.371916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:31.511630 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.511551 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 01:37:31.524602 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.524521 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T01:37:31.511569231Z","UUID":"eb988785-809d-4bad-8c7d-dea19b6f2144","Handler":null,"Name":"","Endpoint":""} Jul 22 01:37:31.526152 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.526135 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 01:37:31.526252 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.526177 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 01:37:31.593137 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.593071 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:31.593137 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.593089 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:31.593404 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.593075 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:31.593404 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:31.593186 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:31.593404 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:31.593259 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:31.593404 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:31.593327 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:31.753805 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.753157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" event={"ID":"5f9f1919-de93-4f46-a439-055b29894833","Type":"ContainerStarted","Data":"995702ab4f880049e85473a19840043427bfa72a78b167005ee7677adf6eb347"} Jul 22 01:37:31.771127 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:31.771079 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ms7nh" podStartSLOduration=3.542830327 podStartE2EDuration="23.771058259s" totalImagesPullingTime="20.228227932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:30.811508046 +0000 UTC m=+25.308949404" watchObservedRunningTime="2026-07-22 01:37:31.771058259 +0000 UTC m=+26.268499617" Jul 22 01:37:32.542981 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:32.542947 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:32.643122 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:32.643085 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:32.757356 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:32.757325 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" event={"ID":"4e7b3b1ca11b8cca62c69eddaf83e8b0","Type":"ContainerStarted","Data":"70b573d87404b8db958fa2a6874280faf37a1cc6cdefd03dccc58c694afbd4cd"} Jul 22 01:37:32.760815 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:32.760795 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:37:32.761260 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:32.761201 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"6588c2d7ec28f02cfbdd18b7aaef8d5374e2e85a6b9c78844884a7f0ca32668b"} Jul 22 01:37:32.774250 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:32.774186 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-174.ec2.internal" podStartSLOduration=24.774169569 podStartE2EDuration="24.774169569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:32.773815912 +0000 UTC m=+27.271257270" watchObservedRunningTime="2026-07-22 01:37:32.774169569 +0000 UTC m=+27.271610928" Jul 22 01:37:33.164914 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:33.164832 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:33.265353 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:33.265313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:33.592707 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:33.592628 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:33.592707 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:33.592666 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:33.592928 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:33.592640 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:33.592928 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:33.592761 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:33.592928 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:33.592838 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:33.592928 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:33.592895 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:33.766264 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:33.766233 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" event={"ID":"5f9f1919-de93-4f46-a439-055b29894833","Type":"ContainerStarted","Data":"456eb97d674242fc80a392f8f565bc580912525b411d202ef3e7e58a08fe0def"} Jul 22 01:37:33.786714 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:33.786662 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ph24p" podStartSLOduration=3.347663998 podStartE2EDuration="25.786644544s" totalImagesPullingTime="22.438980546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:33.786148657 +0000 UTC m=+28.283590016" watchObservedRunningTime="2026-07-22 01:37:33.786644544 +0000 UTC m=+28.284085902" Jul 22 01:37:35.179478 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.179263 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:35.180086 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.179855 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:35.593513 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.593432 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:35.593682 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.593433 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:35.593682 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.593565 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:35.593682 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:35.593560 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:35.593844 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:35.593697 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:35.593904 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:35.593837 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:35.774604 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.774577 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:37:35.774979 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.774957 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"e8b480cffa30eba2d27fadba90d4abbfd97ad0203cc7353b83ea3c0593c709bc"} Jul 22 01:37:35.775295 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.775249 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:35.775398 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.775318 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:35.775467 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.775443 2570 scope.go:117] "RemoveContainer" containerID="0d153937bfba5fc889f814c15758ec56c500417d2848326d6e5d7099f04db373" Jul 22 01:37:35.776920 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.776899 2570 generic.go:356] "Generic (PLEG): container finished" podID="094e4ffe-4ef6-49e6-baf9-8459b8e67813" containerID="d5c904b23b7d12c0a269ca0ce47cb0eb4fcb2d3a4f78135e9b815e037cc9af01" exitCode=0 Jul 22 01:37:35.776997 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.776979 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" event={"ID":"094e4ffe-4ef6-49e6-baf9-8459b8e67813","Type":"ContainerDied","Data":"d5c904b23b7d12c0a269ca0ce47cb0eb4fcb2d3a4f78135e9b815e037cc9af01"} Jul 22 01:37:35.777139 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.777120 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:35.777650 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.777616 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5nkpp" Jul 22 01:37:35.793067 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.793050 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:35.904929 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:35.904903 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:36.103768 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:36.103594 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:36.165829 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:36.165724 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:36.463816 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:36.463731 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:36.783458 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:36.783393 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:37:36.783779 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:36.783756 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" event={"ID":"0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082","Type":"ContainerStarted","Data":"8cc746d596025ee626196b01a8dbd523d7b36218bc10fc24f331392753692f7b"} Jul 22 01:37:36.786708 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:36.785139 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:36.802664 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:36.802636 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:37:36.818898 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:36.818838 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" podStartSLOduration=8.037175011 podStartE2EDuration="28.818816103s" totalImagesPullingTime="20.781641092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:36.818137499 +0000 UTC m=+31.315578857" watchObservedRunningTime="2026-07-22 01:37:36.818816103 +0000 UTC m=+31.316257453" Jul 22 01:37:37.066552 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:37.066521 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7tb7b"] Jul 22 01:37:37.066698 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:37.066654 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:37.066773 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:37.066737 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:37.069400 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:37.069377 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mx4nn"] Jul 22 01:37:37.069524 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:37.069469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:37.069583 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:37.069564 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:37.069969 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:37.069947 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rpm7t"] Jul 22 01:37:37.070083 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:37.070020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:37.070147 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:37.070089 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:37.788016 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:37.787983 2570 generic.go:356] "Generic (PLEG): container finished" podID="094e4ffe-4ef6-49e6-baf9-8459b8e67813" containerID="4b100f8b639592c5b86353640d4f65ba5a94a57b2368ae703183c3ccdf8dec8f" exitCode=0 Jul 22 01:37:37.788509 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:37.788068 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" event={"ID":"094e4ffe-4ef6-49e6-baf9-8459b8e67813","Type":"ContainerDied","Data":"4b100f8b639592c5b86353640d4f65ba5a94a57b2368ae703183c3ccdf8dec8f"} Jul 22 01:37:38.593029 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:38.592965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:38.593029 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:38.592982 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:38.593195 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:38.593069 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:38.593195 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:38.593082 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:38.593195 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:38.593156 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:38.593314 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:38.593208 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:39.660176 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:39.659968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:39.660176 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:39.660173 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:39.660678 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:39.660115 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:39.660678 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:39.660264 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:39.660678 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:39.660283 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs podName:541f80c8-3bbc-40d7-af2c-9d386d3b946c nodeName:}" failed. No retries permitted until 2026-07-22 01:38:11.660262287 +0000 UTC m=+66.157703638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs") pod "network-metrics-daemon-mx4nn" (UID: "541f80c8-3bbc-40d7-af2c-9d386d3b946c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:39.660678 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:39.660303 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret podName:21b6ba48-2cfc-4b03-a77d-0ced99379d04 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:11.660292801 +0000 UTC m=+66.157734137 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret") pod "global-pull-secret-syncer-7tb7b" (UID: "21b6ba48-2cfc-4b03-a77d-0ced99379d04") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:39.761367 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:39.761329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:39.761540 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:39.761459 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:39.761540 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:39.761473 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:39.761540 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:39.761482 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2m4rn for pod openshift-network-diagnostics/network-check-target-rpm7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:39.761540 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:39.761530 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn podName:f636171e-37d9-4c74-bd11-512f7ac26153 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:11.761517818 +0000 UTC m=+66.258959167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2m4rn" (UniqueName: "kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn") pod "network-check-target-rpm7t" (UID: "f636171e-37d9-4c74-bd11-512f7ac26153") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:39.794855 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:39.794818 2570 generic.go:356] "Generic (PLEG): container finished" podID="094e4ffe-4ef6-49e6-baf9-8459b8e67813" containerID="360874b8d79fdcb56d9c52052b16bb325ccf13d769a3f73a15857ae6cc9c3071" exitCode=0 Jul 22 01:37:39.795029 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:39.794862 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" event={"ID":"094e4ffe-4ef6-49e6-baf9-8459b8e67813","Type":"ContainerDied","Data":"360874b8d79fdcb56d9c52052b16bb325ccf13d769a3f73a15857ae6cc9c3071"} Jul 22 01:37:40.536433 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:40.536395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:40.593350 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:40.593319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:40.593350 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:40.593334 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:40.593566 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:40.593319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:40.593566 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:40.593429 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rpm7t" podUID="f636171e-37d9-4c74-bd11-512f7ac26153" Jul 22 01:37:40.593566 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:40.593529 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx4nn" podUID="541f80c8-3bbc-40d7-af2c-9d386d3b946c" Jul 22 01:37:40.593716 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:40.593617 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tb7b" podUID="21b6ba48-2cfc-4b03-a77d-0ced99379d04" Jul 22 01:37:40.611935 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:40.611904 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:40.712691 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:40.712658 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:42.317354 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.317275 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-174.ec2.internal" event="NodeReady" Jul 22 01:37:42.317793 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.317435 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 01:37:42.365440 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.365409 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c4qqk"] Jul 22 01:37:42.367951 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.367926 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mkp5b"] Jul 22 01:37:42.368088 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.368039 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.370461 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.370438 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69f4f59869-vzgnb"] Jul 22 01:37:42.370563 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.370493 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 01:37:42.370563 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.370504 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhl28\"" Jul 22 01:37:42.370751 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.370734 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 01:37:42.370840 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.370826 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mkp5b" Jul 22 01:37:42.373082 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.372917 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v8ltf\"" Jul 22 01:37:42.373082 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.372945 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 01:37:42.373082 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.373007 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 01:37:42.373082 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.373025 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 01:37:42.373392 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.373147 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.375323 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.375252 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fnkgz\"" Jul 22 01:37:42.375428 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.375367 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 01:37:42.376021 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.375872 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 01:37:42.376021 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.375935 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 01:37:42.376886 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.376836 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mkp5b"] Jul 22 01:37:42.379755 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.379734 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4qqk"] Jul 22 01:37:42.379990 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.379893 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ed4451e-6eb6-43e2-b514-daae87017ad2-cert\") pod \"ingress-canary-mkp5b\" (UID: \"2ed4451e-6eb6-43e2-b514-daae87017ad2\") " pod="openshift-ingress-canary/ingress-canary-mkp5b" Jul 22 01:37:42.379990 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.379933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da3008a4-466f-4a75-8a24-483ab8cf6f66-config-volume\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.379990 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.379960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252tt\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-kube-api-access-252tt\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.380198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.379994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.380198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-certificates\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.380198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380051 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-trusted-ca\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.380198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380074 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ck5\" (UniqueName: \"kubernetes.io/projected/da3008a4-466f-4a75-8a24-483ab8cf6f66-kube-api-access-97ck5\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.380198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-image-registry-private-configuration\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.380198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-bound-sa-token\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.380198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.380551 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-installation-pull-secrets\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.380551 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380279 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-ca-trust-extracted\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.380551 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9sb\" (UniqueName: \"kubernetes.io/projected/2ed4451e-6eb6-43e2-b514-daae87017ad2-kube-api-access-vl9sb\") pod \"ingress-canary-mkp5b\" (UID: \"2ed4451e-6eb6-43e2-b514-daae87017ad2\") " pod="openshift-ingress-canary/ingress-canary-mkp5b" Jul 22 01:37:42.380551 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.380347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3008a4-466f-4a75-8a24-483ab8cf6f66-tmp-dir\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.382424 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.381975 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 01:37:42.383176 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.383130 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f4f59869-vzgnb"] Jul 22 01:37:42.481242 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-image-registry-private-configuration\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.481242 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-bound-sa-token\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.481458 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.481458 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-installation-pull-secrets\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.481458 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-ca-trust-extracted\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.481458 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9sb\" (UniqueName: \"kubernetes.io/projected/2ed4451e-6eb6-43e2-b514-daae87017ad2-kube-api-access-vl9sb\") pod \"ingress-canary-mkp5b\" (UID: \"2ed4451e-6eb6-43e2-b514-daae87017ad2\") " pod="openshift-ingress-canary/ingress-canary-mkp5b" Jul 22 01:37:42.481458 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3008a4-466f-4a75-8a24-483ab8cf6f66-tmp-dir\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.481458 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481446 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ed4451e-6eb6-43e2-b514-daae87017ad2-cert\") pod \"ingress-canary-mkp5b\" (UID: \"2ed4451e-6eb6-43e2-b514-daae87017ad2\") " pod="openshift-ingress-canary/ingress-canary-mkp5b" Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481471 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da3008a4-466f-4a75-8a24-483ab8cf6f66-config-volume\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.481477 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481497 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252tt\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-kube-api-access-252tt\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.481561 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls podName:da3008a4-466f-4a75-8a24-483ab8cf6f66 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:42.981537553 +0000 UTC m=+37.478978888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls") pod "dns-default-c4qqk" (UID: "da3008a4-466f-4a75-8a24-483ab8cf6f66") : secret "dns-default-metrics-tls" not found Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.481614 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481614 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-certificates\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.481628 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f4f59869-vzgnb: secret "image-registry-tls" not found Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-trusted-ca\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.481672 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls podName:e2b14a37-cbd1-4987-90fe-7348f0b72b1a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:42.981657004 +0000 UTC m=+37.479098341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls") pod "image-registry-69f4f59869-vzgnb" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a") : secret "image-registry-tls" not found Jul 22 01:37:42.481755 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481690 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ck5\" (UniqueName: \"kubernetes.io/projected/da3008a4-466f-4a75-8a24-483ab8cf6f66-kube-api-access-97ck5\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.482345 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.481836 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3008a4-466f-4a75-8a24-483ab8cf6f66-tmp-dir\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.482345 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.482146 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-ca-trust-extracted\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.482633 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.482607 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da3008a4-466f-4a75-8a24-483ab8cf6f66-config-volume\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.482768 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.482641 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-certificates\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.482768 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.482645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-trusted-ca\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.486577 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.486553 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-installation-pull-secrets\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.486684 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.486590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ed4451e-6eb6-43e2-b514-daae87017ad2-cert\") pod \"ingress-canary-mkp5b\" (UID: \"2ed4451e-6eb6-43e2-b514-daae87017ad2\") " pod="openshift-ingress-canary/ingress-canary-mkp5b" Jul 22 01:37:42.486684 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.486670 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-image-registry-private-configuration\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.492285 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.492260 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ck5\" (UniqueName: \"kubernetes.io/projected/da3008a4-466f-4a75-8a24-483ab8cf6f66-kube-api-access-97ck5\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.492423 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.492397 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-252tt\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-kube-api-access-252tt\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.492502 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.492419 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9sb\" (UniqueName: \"kubernetes.io/projected/2ed4451e-6eb6-43e2-b514-daae87017ad2-kube-api-access-vl9sb\") pod \"ingress-canary-mkp5b\" (UID: \"2ed4451e-6eb6-43e2-b514-daae87017ad2\") " pod="openshift-ingress-canary/ingress-canary-mkp5b" Jul 22 01:37:42.492772 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.492744 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-bound-sa-token\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.516786 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.516743 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:42.592490 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.592411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:37:42.592634 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.592429 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:37:42.592634 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.592603 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:37:42.595135 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.595106 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fbrqb\"" Jul 22 01:37:42.595135 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.595135 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 01:37:42.595333 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.595320 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 01:37:42.595429 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.595409 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 01:37:42.595753 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.595439 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqzqp\"" Jul 22 01:37:42.595883 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.595865 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 01:37:42.614705 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.614679 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:42.690186 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.690157 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mkp5b" Jul 22 01:37:42.986006 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.985922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:42.986006 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:42.985992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:42.986250 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.986075 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:42.986250 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.986147 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls podName:da3008a4-466f-4a75-8a24-483ab8cf6f66 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:43.986131396 +0000 UTC m=+38.483572748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls") pod "dns-default-c4qqk" (UID: "da3008a4-466f-4a75-8a24-483ab8cf6f66") : secret "dns-default-metrics-tls" not found Jul 22 01:37:42.986250 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.986078 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 01:37:42.986250 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.986205 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f4f59869-vzgnb: secret "image-registry-tls" not found Jul 22 01:37:42.986460 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:42.986273 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls podName:e2b14a37-cbd1-4987-90fe-7348f0b72b1a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:43.986257831 +0000 UTC m=+38.483699166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls") pod "image-registry-69f4f59869-vzgnb" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a") : secret "image-registry-tls" not found Jul 22 01:37:43.993696 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:43.993651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:43.994382 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:43.993767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:43.994382 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:43.993806 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:43.994382 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:43.993877 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls podName:da3008a4-466f-4a75-8a24-483ab8cf6f66 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:45.993861182 +0000 UTC m=+40.491302521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls") pod "dns-default-c4qqk" (UID: "da3008a4-466f-4a75-8a24-483ab8cf6f66") : secret "dns-default-metrics-tls" not found Jul 22 01:37:43.994382 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:43.993893 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 01:37:43.994382 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:43.993906 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f4f59869-vzgnb: secret "image-registry-tls" not found Jul 22 01:37:43.994382 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:43.993970 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls podName:e2b14a37-cbd1-4987-90fe-7348f0b72b1a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:45.993954028 +0000 UTC m=+40.491395381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls") pod "image-registry-69f4f59869-vzgnb" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a") : secret "image-registry-tls" not found Jul 22 01:37:44.108176 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:44.108138 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:44.166747 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:44.166601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:44.252813 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:44.252633 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mkp5b"] Jul 22 01:37:44.258251 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:44.258224 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed4451e_6eb6_43e2_b514_daae87017ad2.slice/crio-cf85f4855ea4f23be87738662f1688765547fdcf8687554822520f6f1934394f WatchSource:0}: Error finding container cf85f4855ea4f23be87738662f1688765547fdcf8687554822520f6f1934394f: Status 404 returned error can't find the container with id cf85f4855ea4f23be87738662f1688765547fdcf8687554822520f6f1934394f Jul 22 01:37:44.455833 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:44.455801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:44.510742 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:44.510665 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:44.809630 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:44.809548 2570 generic.go:356] "Generic (PLEG): container finished" podID="094e4ffe-4ef6-49e6-baf9-8459b8e67813" containerID="d4b5c0c024d034411ff9589362ea708216bcee65c8fc3dc0d7e1a18b26950886" exitCode=0 Jul 22 01:37:44.809804 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:44.809639 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" event={"ID":"094e4ffe-4ef6-49e6-baf9-8459b8e67813","Type":"ContainerDied","Data":"d4b5c0c024d034411ff9589362ea708216bcee65c8fc3dc0d7e1a18b26950886"} Jul 22 01:37:44.810626 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:44.810605 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mkp5b" event={"ID":"2ed4451e-6eb6-43e2-b514-daae87017ad2","Type":"ContainerStarted","Data":"cf85f4855ea4f23be87738662f1688765547fdcf8687554822520f6f1934394f"} Jul 22 01:37:45.817869 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:45.817839 2570 generic.go:356] "Generic (PLEG): container finished" podID="094e4ffe-4ef6-49e6-baf9-8459b8e67813" containerID="6a08239250e7faf3f7f9587b6e2b1084b473f7bf5c89137736e1b19c51e5aa00" exitCode=0 Jul 22 01:37:45.818348 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:45.817884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" event={"ID":"094e4ffe-4ef6-49e6-baf9-8459b8e67813","Type":"ContainerDied","Data":"6a08239250e7faf3f7f9587b6e2b1084b473f7bf5c89137736e1b19c51e5aa00"} Jul 22 01:37:46.010444 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:46.010400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:46.010623 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:46.010518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:46.010623 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:46.010610 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 01:37:46.010747 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:46.010636 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f4f59869-vzgnb: secret "image-registry-tls" not found Jul 22 01:37:46.010747 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:46.010671 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:46.010747 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:46.010714 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls podName:e2b14a37-cbd1-4987-90fe-7348f0b72b1a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:50.010690989 +0000 UTC m=+44.508132343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls") pod "image-registry-69f4f59869-vzgnb" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a") : secret "image-registry-tls" not found Jul 22 01:37:46.010747 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:46.010732 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls podName:da3008a4-466f-4a75-8a24-483ab8cf6f66 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:50.010723601 +0000 UTC m=+44.508164944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls") pod "dns-default-c4qqk" (UID: "da3008a4-466f-4a75-8a24-483ab8cf6f66") : secret "dns-default-metrics-tls" not found Jul 22 01:37:46.823882 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:46.823848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" event={"ID":"094e4ffe-4ef6-49e6-baf9-8459b8e67813","Type":"ContainerStarted","Data":"0e0e0e4f592b6bb5a130ccb63a670938c1a374e79436e494caebf58fcbbf0566"} Jul 22 01:37:46.850956 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:46.850899 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5lvdv" podStartSLOduration=7.35069625 podStartE2EDuration="38.850871422s" totalImagesPullingTime="31.500175172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:46.849362587 +0000 UTC m=+41.346803962" watchObservedRunningTime="2026-07-22 01:37:46.850871422 +0000 UTC m=+41.348312781" Jul 22 01:37:46.886322 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:46.886247 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:46.974350 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:46.974325 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:47.562595 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:47.562552 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:47.827570 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:47.827489 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mkp5b" event={"ID":"2ed4451e-6eb6-43e2-b514-daae87017ad2","Type":"ContainerStarted","Data":"861fc0693f9cd75965b678cf0eabcfac8a68e16f3701a134d92e27359be5c813"} Jul 22 01:37:47.843944 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:47.843892 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mkp5b" podStartSLOduration=3.174158104 podStartE2EDuration="5.84387522s" totalImagesPullingTime="2.669717116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:47.843338141 +0000 UTC m=+42.340779498" watchObservedRunningTime="2026-07-22 01:37:47.84387522 +0000 UTC m=+42.341316577" Jul 22 01:37:48.264669 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.264631 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:48.364565 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.364517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:48.695276 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.695241 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:48.698512 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.698486 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rh9g2"] Jul 22 01:37:48.740570 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.740544 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rh9g2"] Jul 22 01:37:48.740695 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.740638 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.742944 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.742918 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 01:37:48.743076 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.742999 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 01:37:48.743140 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.743098 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 01:37:48.743382 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.743207 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 01:37:48.743470 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.743276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l77sq\"" Jul 22 01:37:48.768583 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.768548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:48.833956 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.833927 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-data-volume\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.834325 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.833998 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.834325 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.834028 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-crio-socket\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.834325 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.834052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.834325 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.834111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ddc\" (UniqueName: \"kubernetes.io/projected/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-api-access-h7ddc\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.903493 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.903459 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:48.934958 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.934932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-data-volume\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.935079 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.935042 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.935119 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.935096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-crio-socket\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.935153 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.935121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.935153 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.935145 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ddc\" (UniqueName: \"kubernetes.io/projected/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-api-access-h7ddc\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.935273 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:48.935229 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:48.935322 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:48.935276 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:48.935322 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:48.935301 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:49.435281982 +0000 UTC m=+43.932723328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : configmap "kube-rbac-proxy" not found Jul 22 01:37:48.935415 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:48.935339 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:49.435322763 +0000 UTC m=+43.932764122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:48.935415 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.935350 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-crio-socket\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.935517 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.935499 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-data-volume\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:48.951043 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:48.950991 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ddc\" (UniqueName: \"kubernetes.io/projected/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-api-access-h7ddc\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:49.004475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:49.004446 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:49.439486 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:49.439448 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:49.439663 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:49.439512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:49.439663 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:49.439573 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:49.439663 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:49.439608 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:49.439663 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:49.439650 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:50.439633409 +0000 UTC m=+44.937074766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : configmap "kube-rbac-proxy" not found Jul 22 01:37:49.439809 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:49.439669 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:50.43965727 +0000 UTC m=+44.937098604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:49.840796 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:49.840698 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:49.898368 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:49.898333 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:50.043919 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:50.043881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:50.044104 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:50.043953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:50.044104 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.044058 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 01:37:50.044104 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.044079 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f4f59869-vzgnb: secret "image-registry-tls" not found Jul 22 01:37:50.044104 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.044096 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:50.044323 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.044160 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls podName:e2b14a37-cbd1-4987-90fe-7348f0b72b1a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:58.044140393 +0000 UTC m=+52.541581751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls") pod "image-registry-69f4f59869-vzgnb" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a") : secret "image-registry-tls" not found Jul 22 01:37:50.044323 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.044182 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls podName:da3008a4-466f-4a75-8a24-483ab8cf6f66 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:58.044173006 +0000 UTC m=+52.541614344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls") pod "dns-default-c4qqk" (UID: "da3008a4-466f-4a75-8a24-483ab8cf6f66") : secret "dns-default-metrics-tls" not found Jul 22 01:37:50.446528 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:50.446486 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:50.446732 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:50.446605 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:50.446732 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.446643 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:50.446732 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.446686 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:50.446732 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.446715 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:52.446699303 +0000 UTC m=+46.944140658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:50.446900 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:50.446737 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:52.446722544 +0000 UTC m=+46.944163879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : configmap "kube-rbac-proxy" not found Jul 22 01:37:50.789568 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:50.789499 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ms7nh_9868bf39-940c-4e86-9405-1920fc525b8a/dns-node-resolver/0.log" Jul 22 01:37:51.163057 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.163025 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-6jmqq"] Jul 22 01:37:51.185782 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.185759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-6jmqq"] Jul 22 01:37:51.185902 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.185847 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.188206 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.188183 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 22 01:37:51.189132 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.189110 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 22 01:37:51.189132 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.189124 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fhh7t\"" Jul 22 01:37:51.189316 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.189152 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 22 01:37:51.189316 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.189202 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 22 01:37:51.205782 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.205754 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:51.253424 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.253399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/882e79e4-69fd-4051-b95b-737e92336591-signing-cabundle\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.253506 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.253445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfqv9\" (UniqueName: \"kubernetes.io/projected/882e79e4-69fd-4051-b95b-737e92336591-kube-api-access-gfqv9\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.253560 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.253539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/882e79e4-69fd-4051-b95b-737e92336591-signing-key\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.258760 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.258736 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:51.354690 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.354646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/882e79e4-69fd-4051-b95b-737e92336591-signing-cabundle\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.354848 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.354813 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfqv9\" (UniqueName: \"kubernetes.io/projected/882e79e4-69fd-4051-b95b-737e92336591-kube-api-access-gfqv9\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.354919 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.354905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/882e79e4-69fd-4051-b95b-737e92336591-signing-key\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.355499 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.355480 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/882e79e4-69fd-4051-b95b-737e92336591-signing-cabundle\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.357191 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.357170 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/882e79e4-69fd-4051-b95b-737e92336591-signing-key\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.363669 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.363645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfqv9\" (UniqueName: \"kubernetes.io/projected/882e79e4-69fd-4051-b95b-737e92336591-kube-api-access-gfqv9\") pod \"service-ca-58598695f7-6jmqq\" (UID: \"882e79e4-69fd-4051-b95b-737e92336591\") " pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.494900 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.494795 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-6jmqq" Jul 22 01:37:51.658946 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.658911 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-6jmqq"] Jul 22 01:37:51.661607 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:37:51.661584 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882e79e4_69fd_4051_b95b_737e92336591.slice/crio-a2abfeb2ef3e574cfa9c68c2a38f2d8b97bb31df036c42e4078d92722180585c WatchSource:0}: Error finding container a2abfeb2ef3e574cfa9c68c2a38f2d8b97bb31df036c42e4078d92722180585c: Status 404 returned error can't find the container with id a2abfeb2ef3e574cfa9c68c2a38f2d8b97bb31df036c42e4078d92722180585c Jul 22 01:37:51.795581 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.795513 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvxs5_aebe006d-9a1d-4049-b3e1-37130f7339aa/node-ca/0.log" Jul 22 01:37:51.836430 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:51.836400 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-6jmqq" event={"ID":"882e79e4-69fd-4051-b95b-737e92336591","Type":"ContainerStarted","Data":"a2abfeb2ef3e574cfa9c68c2a38f2d8b97bb31df036c42e4078d92722180585c"} Jul 22 01:37:52.023151 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:52.023116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:52.462591 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:52.462559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:52.463033 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:52.462619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:52.463033 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:52.462703 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:52.463033 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:52.462748 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:52.463033 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:52.462771 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:56.462755178 +0000 UTC m=+50.960196518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : configmap "kube-rbac-proxy" not found Jul 22 01:37:52.463033 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:52.462809 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:56.462789845 +0000 UTC m=+50.960231181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:52.590809 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:52.590750 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mkp5b_2ed4451e-6eb6-43e2-b514-daae87017ad2/serve-healthcheck-canary/0.log" Jul 22 01:37:54.848307 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:54.848269 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-6jmqq" event={"ID":"882e79e4-69fd-4051-b95b-737e92336591","Type":"ContainerStarted","Data":"6e5f84008793a88c1b740ff8c27bf12470041467759e79fa2d1df1be4f01623a"} Jul 22 01:37:54.873047 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:54.872998 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-6jmqq" podStartSLOduration=1.645372669 podStartE2EDuration="3.872982647s" totalImagesPullingTime="2.227609978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:54.871628066 +0000 UTC m=+49.369069423" watchObservedRunningTime="2026-07-22 01:37:54.872982647 +0000 UTC m=+49.370424057" Jul 22 01:37:55.140953 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:55.140907 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:55.346505 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:55.346472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:55.445974 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:55.445891 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:55.562742 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:55.562698 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:56.030326 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:56.030283 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:56.427018 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:56.426984 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:56.494588 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:56.494555 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:56.494755 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:56.494605 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:37:56.494755 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:56.494689 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:56.494755 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:56.494711 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:56.494755 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:56.494750 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:04.494737503 +0000 UTC m=+58.992178838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:56.495027 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:56.494763 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:04.494756196 +0000 UTC m=+58.992197530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : configmap "kube-rbac-proxy" not found Jul 22 01:37:57.061173 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:57.061120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:57.437031 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:57.436977 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:57.842302 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:57.842200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:58.042805 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:58.042769 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:58.106923 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:58.106901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:37:58.107287 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:58.106978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:37:58.107287 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:58.107053 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:58.107287 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:58.107077 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 01:37:58.107287 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:58.107089 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f4f59869-vzgnb: secret "image-registry-tls" not found Jul 22 01:37:58.107287 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:58.107110 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls podName:da3008a4-466f-4a75-8a24-483ab8cf6f66 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:14.1070957 +0000 UTC m=+68.604537035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls") pod "dns-default-c4qqk" (UID: "da3008a4-466f-4a75-8a24-483ab8cf6f66") : secret "dns-default-metrics-tls" not found Jul 22 01:37:58.107287 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:37:58.107130 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls podName:e2b14a37-cbd1-4987-90fe-7348f0b72b1a nodeName:}" failed. No retries permitted until 2026-07-22 01:38:14.107118484 +0000 UTC m=+68.604559823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls") pod "image-registry-69f4f59869-vzgnb" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a") : secret "image-registry-tls" not found Jul 22 01:37:58.450376 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:58.450302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:58.841381 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:58.841289 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:59.207893 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:59.207862 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:59.287473 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:59.287447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:59.676187 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:59.676144 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:59.834598 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:59.834563 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:37:59.889849 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:37:59.889813 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:00.239765 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:00.239720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:00.436002 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:00.435971 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:00.638791 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:00.638757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:01.466644 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:01.466608 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:01.829767 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:01.829687 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:02.025664 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:02.025613 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:02.235920 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:02.235881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:02.561446 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:02.561353 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:02.664934 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:02.664897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:02.968681 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:02.968645 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:03.829459 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:03.829425 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:03.930622 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:03.930590 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:04.033962 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:04.033931 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:04.253451 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:04.253419 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:04.435381 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:04.435348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:04.557498 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:04.557420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:38:04.557498 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:04.557475 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:38:04.557651 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:38:04.557574 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:38:04.557651 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:38:04.557632 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls podName:c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:20.557619851 +0000 UTC m=+75.055061186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rh9g2" (UID: "c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18") : secret "insights-runtime-extractor-tls" not found Jul 22 01:38:04.557933 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:04.557914 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:38:04.828094 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:04.828012 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:05.432782 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:05.432749 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:05.623793 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:05.623757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:05.828652 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:05.828570 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:06.106605 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:06.106573 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:06.175939 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:06.175906 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:07.398140 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:07.398107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:07.499538 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:07.499505 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:07.557163 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:07.557120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:07.689438 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:07.689362 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:08.055053 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:08.054978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:08.108829 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:08.108803 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:08.497821 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:08.497789 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:08.556557 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:08.556517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:08.808005 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:08.807940 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jgthx" Jul 22 01:38:09.269767 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:09.269736 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:09.468738 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:09.468692 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:09.575360 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:09.575279 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:11.714586 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.714531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:38:11.715096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.714600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:38:11.716909 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.716892 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 01:38:11.717598 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.717583 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 01:38:11.727694 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.727671 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21b6ba48-2cfc-4b03-a77d-0ced99379d04-original-pull-secret\") pod \"global-pull-secret-syncer-7tb7b\" (UID: \"21b6ba48-2cfc-4b03-a77d-0ced99379d04\") " pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:38:11.727837 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.727822 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541f80c8-3bbc-40d7-af2c-9d386d3b946c-metrics-certs\") pod \"network-metrics-daemon-mx4nn\" (UID: \"541f80c8-3bbc-40d7-af2c-9d386d3b946c\") " pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:38:11.815584 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.815531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:38:11.818157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.818137 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 01:38:11.828252 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.828230 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 01:38:11.839422 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:11.839394 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4rn\" (UniqueName: \"kubernetes.io/projected/f636171e-37d9-4c74-bd11-512f7ac26153-kube-api-access-2m4rn\") pod \"network-check-target-rpm7t\" (UID: \"f636171e-37d9-4c74-bd11-512f7ac26153\") " pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:38:12.008611 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.008536 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fbrqb\"" Jul 22 01:38:12.013444 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.013424 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tb7b" Jul 22 01:38:12.016492 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.016470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx4nn" Jul 22 01:38:12.022853 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.022834 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqzqp\"" Jul 22 01:38:12.031720 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.031701 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:38:12.172184 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.172140 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7tb7b"] Jul 22 01:38:12.173455 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.173368 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mx4nn"] Jul 22 01:38:12.175087 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:38:12.175063 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541f80c8_3bbc_40d7_af2c_9d386d3b946c.slice/crio-973a362268097ce99da8b857fbfe2ac83ece9935fff00d8456aed1f47941fd44 WatchSource:0}: Error finding container 973a362268097ce99da8b857fbfe2ac83ece9935fff00d8456aed1f47941fd44: Status 404 returned error can't find the container with id 973a362268097ce99da8b857fbfe2ac83ece9935fff00d8456aed1f47941fd44 Jul 22 01:38:12.175482 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:38:12.175468 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b6ba48_2cfc_4b03_a77d_0ced99379d04.slice/crio-ead04cf234c428a55fc95ba66513f458bf20ba64764bf3a2d1f0a8a299402311 WatchSource:0}: Error finding container ead04cf234c428a55fc95ba66513f458bf20ba64764bf3a2d1f0a8a299402311: Status 404 returned error can't find the container with id ead04cf234c428a55fc95ba66513f458bf20ba64764bf3a2d1f0a8a299402311 Jul 22 01:38:12.188887 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.188861 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rpm7t"] Jul 22 01:38:12.191168 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:38:12.191150 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf636171e_37d9_4c74_bd11_512f7ac26153.slice/crio-c3dcc15b068be6cd1ccd076b43bb6cabb81b12ad8f7f2cf4ea10a58271aecc83 WatchSource:0}: Error finding container c3dcc15b068be6cd1ccd076b43bb6cabb81b12ad8f7f2cf4ea10a58271aecc83: Status 404 returned error can't find the container with id c3dcc15b068be6cd1ccd076b43bb6cabb81b12ad8f7f2cf4ea10a58271aecc83 Jul 22 01:38:12.891091 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.891050 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx4nn" event={"ID":"541f80c8-3bbc-40d7-af2c-9d386d3b946c","Type":"ContainerStarted","Data":"973a362268097ce99da8b857fbfe2ac83ece9935fff00d8456aed1f47941fd44"} Jul 22 01:38:12.892317 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.892278 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rpm7t" event={"ID":"f636171e-37d9-4c74-bd11-512f7ac26153","Type":"ContainerStarted","Data":"c3dcc15b068be6cd1ccd076b43bb6cabb81b12ad8f7f2cf4ea10a58271aecc83"} Jul 22 01:38:12.893461 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.893438 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7tb7b" event={"ID":"21b6ba48-2cfc-4b03-a77d-0ced99379d04","Type":"ContainerStarted","Data":"ead04cf234c428a55fc95ba66513f458bf20ba64764bf3a2d1f0a8a299402311"} Jul 22 01:38:12.903327 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.903237 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:12.958156 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:12.958119 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:13.899552 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:13.899517 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx4nn" event={"ID":"541f80c8-3bbc-40d7-af2c-9d386d3b946c","Type":"ContainerStarted","Data":"62813a089ad4c63b4003c1901e312d69bec4293bc981dbdab39effe9285ae466"} Jul 22 01:38:14.010126 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.010087 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7"] Jul 22 01:38:14.013055 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.013032 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f4f59869-vzgnb"] Jul 22 01:38:14.013170 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.013069 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn"] Jul 22 01:38:14.013248 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.013164 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.013429 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:38:14.013367 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" podUID="e2b14a37-cbd1-4987-90fe-7348f0b72b1a" Jul 22 01:38:14.015548 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.015529 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 01:38:14.015650 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.015629 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 01:38:14.015709 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.015689 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 01:38:14.015764 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.015702 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 01:38:14.016151 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.016135 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.018629 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.018611 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 22 01:38:14.018734 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.018635 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 22 01:38:14.018734 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.018645 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 22 01:38:14.018734 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.018706 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 22 01:38:14.033102 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.033082 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7"] Jul 22 01:38:14.038450 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.038432 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn"] Jul 22 01:38:14.115261 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.115193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:14.137058 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:38:14.137279 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-tmp\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.137279 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137129 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:38:14.137279 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.137279 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137180 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.137279 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e034cd3d-685e-4594-85e8-413cb2b330a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.137279 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137244 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-hub\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.137279 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8hq\" (UniqueName: \"kubernetes.io/projected/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-kube-api-access-sc8hq\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.137650 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.137650 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmm45\" (UniqueName: \"kubernetes.io/projected/e034cd3d-685e-4594-85e8-413cb2b330a1-kube-api-access-jmm45\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.137650 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.137363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-ca\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.140103 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.140074 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da3008a4-466f-4a75-8a24-483ab8cf6f66-metrics-tls\") pod \"dns-default-c4qqk\" (UID: \"da3008a4-466f-4a75-8a24-483ab8cf6f66\") " pod="openshift-dns/dns-default-c4qqk" Jul 22 01:38:14.140328 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.140308 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"image-registry-69f4f59869-vzgnb\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:38:14.184448 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.184422 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhl28\"" Jul 22 01:38:14.192474 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.192451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4qqk" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.237765 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.237820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmm45\" (UniqueName: \"kubernetes.io/projected/e034cd3d-685e-4594-85e8-413cb2b330a1-kube-api-access-jmm45\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.237847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-ca\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.237883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-tmp\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.237946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.237962 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.237971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.237999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e034cd3d-685e-4594-85e8-413cb2b330a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.238027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-hub\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.238059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8hq\" (UniqueName: \"kubernetes.io/projected/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-kube-api-access-sc8hq\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.238475 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.238412 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-tmp\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.238989 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.238927 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e034cd3d-685e-4594-85e8-413cb2b330a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.241174 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.241128 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-ca\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.241936 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.241887 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.241936 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.241887 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.242649 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.242629 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-hub\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.242783 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.242760 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e034cd3d-685e-4594-85e8-413cb2b330a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.253611 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.253558 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8hq\" (UniqueName: \"kubernetes.io/projected/6af2f41c-898d-48ab-b7b2-ca2214f2d0f7-kube-api-access-sc8hq\") pod \"klusterlet-addon-workmgr-5d8554748f-d4ds7\" (UID: \"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.254176 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.254123 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmm45\" (UniqueName: \"kubernetes.io/projected/e034cd3d-685e-4594-85e8-413cb2b330a1-kube-api-access-jmm45\") pod \"cluster-proxy-proxy-agent-76768768fb-c6ztn\" (UID: \"e034cd3d-685e-4594-85e8-413cb2b330a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.322923 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.322676 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:14.335405 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.335371 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" Jul 22 01:38:14.343812 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.343781 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4qqk"] Jul 22 01:38:14.490866 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.490802 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7"] Jul 22 01:38:14.495205 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:38:14.495176 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af2f41c_898d_48ab_b7b2_ca2214f2d0f7.slice/crio-97019dc16fb62db9c6aa30f0e30a475bbd4507973d57a927112f2ac2da1d3545 WatchSource:0}: Error finding container 97019dc16fb62db9c6aa30f0e30a475bbd4507973d57a927112f2ac2da1d3545: Status 404 returned error can't find the container with id 97019dc16fb62db9c6aa30f0e30a475bbd4507973d57a927112f2ac2da1d3545 Jul 22 01:38:14.510787 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.510727 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn"] Jul 22 01:38:14.514401 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:38:14.514357 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode034cd3d_685e_4594_85e8_413cb2b330a1.slice/crio-5ed52c9ffeec8a136a834cce85b46ae74c3a546a0d0c189dc6f164daafc227f7 WatchSource:0}: Error finding container 5ed52c9ffeec8a136a834cce85b46ae74c3a546a0d0c189dc6f164daafc227f7: Status 404 returned error can't find the container with id 5ed52c9ffeec8a136a834cce85b46ae74c3a546a0d0c189dc6f164daafc227f7 Jul 22 01:38:14.906568 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.906263 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx4nn" event={"ID":"541f80c8-3bbc-40d7-af2c-9d386d3b946c","Type":"ContainerStarted","Data":"45c70003dc98c11136d5126940fbf513cbe8490c0bebe0c9a5d5ebd6c52c3879"} Jul 22 01:38:14.907767 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.907732 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4qqk" event={"ID":"da3008a4-466f-4a75-8a24-483ab8cf6f66","Type":"ContainerStarted","Data":"9f12d97654b533184863b468558a6c2bcd54998afea17d721dfd5f89096c6383"} Jul 22 01:38:14.909541 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.909420 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" event={"ID":"e034cd3d-685e-4594-85e8-413cb2b330a1","Type":"ContainerStarted","Data":"5ed52c9ffeec8a136a834cce85b46ae74c3a546a0d0c189dc6f164daafc227f7"} Jul 22 01:38:14.910930 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.910908 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:38:14.911183 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.911163 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" event={"ID":"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7","Type":"ContainerStarted","Data":"97019dc16fb62db9c6aa30f0e30a475bbd4507973d57a927112f2ac2da1d3545"} Jul 22 01:38:14.918638 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.918521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:38:14.923670 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:14.923622 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mx4nn" podStartSLOduration=65.323757342 podStartE2EDuration="1m6.923602912s" totalImagesPullingTime="1.59984557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:14.922796964 +0000 UTC m=+69.420238344" watchObservedRunningTime="2026-07-22 01:38:14.923602912 +0000 UTC m=+69.421044272" Jul 22 01:38:15.044115 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.043835 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-image-registry-private-configuration\") pod \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " Jul 22 01:38:15.044115 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.043882 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-ca-trust-extracted\") pod \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " Jul 22 01:38:15.044115 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.044014 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-252tt\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-kube-api-access-252tt\") pod \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " Jul 22 01:38:15.044115 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.044072 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-installation-pull-secrets\") pod \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " Jul 22 01:38:15.044115 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.044110 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") pod \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " Jul 22 01:38:15.044545 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.044134 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-trusted-ca\") pod \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " Jul 22 01:38:15.044545 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.044178 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-bound-sa-token\") pod \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " Jul 22 01:38:15.044545 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.044209 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-certificates\") pod \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\" (UID: \"e2b14a37-cbd1-4987-90fe-7348f0b72b1a\") " Jul 22 01:38:15.045014 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.044763 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e2b14a37-cbd1-4987-90fe-7348f0b72b1a" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:38:15.045451 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.045152 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:15.045451 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.045311 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e2b14a37-cbd1-4987-90fe-7348f0b72b1a" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:38:15.045595 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.045543 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e2b14a37-cbd1-4987-90fe-7348f0b72b1a" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:38:15.051525 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.051189 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e2b14a37-cbd1-4987-90fe-7348f0b72b1a" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:38:15.051525 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.051195 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e2b14a37-cbd1-4987-90fe-7348f0b72b1a" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:38:15.051525 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.051502 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e2b14a37-cbd1-4987-90fe-7348f0b72b1a" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:38:15.051876 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.051852 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e2b14a37-cbd1-4987-90fe-7348f0b72b1a" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:38:15.052406 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.052380 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-kube-api-access-252tt" (OuterVolumeSpecName: "kube-api-access-252tt") pod "e2b14a37-cbd1-4987-90fe-7348f0b72b1a" (UID: "e2b14a37-cbd1-4987-90fe-7348f0b72b1a"). InnerVolumeSpecName "kube-api-access-252tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:38:15.140191 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.140152 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:15.145696 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.145669 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-bound-sa-token\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:38:15.145696 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.145696 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-certificates\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:38:15.145917 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.145713 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-image-registry-private-configuration\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:38:15.145917 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.145728 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-ca-trust-extracted\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:38:15.145917 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.145739 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252tt\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-kube-api-access-252tt\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:38:15.145917 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.145747 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-installation-pull-secrets\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:38:15.145917 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.145757 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-registry-tls\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:38:15.145917 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.145769 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b14a37-cbd1-4987-90fe-7348f0b72b1a-trusted-ca\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:38:15.324266 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.324167 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:15.599979 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.599901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:15.866335 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.866254 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:15.913270 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.913235 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f4f59869-vzgnb" Jul 22 01:38:15.947836 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.947808 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f4f59869-vzgnb"] Jul 22 01:38:15.955460 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:15.955430 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69f4f59869-vzgnb"] Jul 22 01:38:16.259875 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:16.259781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:16.359704 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:16.359671 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:17.112457 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:17.112429 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:17.340226 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:17.340178 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:17.596641 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:17.596568 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b14a37-cbd1-4987-90fe-7348f0b72b1a" path="/var/lib/kubelet/pods/e2b14a37-cbd1-4987-90fe-7348f0b72b1a/volumes" Jul 22 01:38:17.693332 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:17.693290 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:17.757796 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:17.757757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:18.935656 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.935191 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rpm7t" event={"ID":"f636171e-37d9-4c74-bd11-512f7ac26153","Type":"ContainerStarted","Data":"840916ad4eb30c93de836d1df70b89ade3b992238ecea5fc0399fac812aab754"} Jul 22 01:38:18.935656 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.935547 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:38:18.938845 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.938383 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7tb7b" event={"ID":"21b6ba48-2cfc-4b03-a77d-0ced99379d04","Type":"ContainerStarted","Data":"78f62e41b3d620cb4b207a3712c6a4af65c905fe7c4725e191a2c0a9dafd653f"} Jul 22 01:38:18.944399 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.944362 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4qqk" event={"ID":"da3008a4-466f-4a75-8a24-483ab8cf6f66","Type":"ContainerStarted","Data":"8b2daf5847219afc197456104168203c5e7dcf995a4fe2bb193afd20fb3aedec"} Jul 22 01:38:18.944525 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.944415 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4qqk" event={"ID":"da3008a4-466f-4a75-8a24-483ab8cf6f66","Type":"ContainerStarted","Data":"54532ccf76e45987b13f93a4d632ac269ce5c279b8a02e9a917698f18c152e61"} Jul 22 01:38:18.945109 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.944764 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c4qqk" Jul 22 01:38:18.953507 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.953082 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rpm7t" podStartSLOduration=64.549124374 podStartE2EDuration="1m10.953064899s" totalImagesPullingTime="6.403940525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:18.952112625 +0000 UTC m=+73.449554007" watchObservedRunningTime="2026-07-22 01:38:18.953064899 +0000 UTC m=+73.450506258" Jul 22 01:38:18.994889 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.994827 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7tb7b" podStartSLOduration=64.575051115 podStartE2EDuration="1m10.994807092s" totalImagesPullingTime="6.419755977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:18.993895162 +0000 UTC m=+73.491336533" watchObservedRunningTime="2026-07-22 01:38:18.994807092 +0000 UTC m=+73.492248450" Jul 22 01:38:18.995367 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:18.995329 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c4qqk" podStartSLOduration=32.744775271 podStartE2EDuration="36.995318084s" totalImagesPullingTime="4.250542813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:18.971913075 +0000 UTC m=+73.469354432" watchObservedRunningTime="2026-07-22 01:38:18.995318084 +0000 UTC m=+73.492759444" Jul 22 01:38:20.589768 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:20.589723 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:38:20.592322 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:20.592291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rh9g2\" (UID: \"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18\") " pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:38:20.852349 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:20.852325 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l77sq\"" Jul 22 01:38:20.860151 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:20.860124 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rh9g2" Jul 22 01:38:20.992905 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:20.992758 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rh9g2"] Jul 22 01:38:20.995861 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:38:20.995835 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cefbba_a9d8_4b50_a0a7_6bff2ef18f18.slice/crio-c70654e3845befca8d33b0ee7212a9fabb94c0e6a0bcad1b35d2a579cbba065b WatchSource:0}: Error finding container c70654e3845befca8d33b0ee7212a9fabb94c0e6a0bcad1b35d2a579cbba065b: Status 404 returned error can't find the container with id c70654e3845befca8d33b0ee7212a9fabb94c0e6a0bcad1b35d2a579cbba065b Jul 22 01:38:21.959425 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:21.959386 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rh9g2" event={"ID":"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18","Type":"ContainerStarted","Data":"c70654e3845befca8d33b0ee7212a9fabb94c0e6a0bcad1b35d2a579cbba065b"} Jul 22 01:38:22.234104 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:22.234015 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:22.298149 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:22.298113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:23.319473 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:23.319443 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:23.388069 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:23.388036 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:23.540732 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:23.540690 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:23.640876 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:23.640836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:24.358708 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:24.358647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:24.422403 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:24.422357 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:24.601932 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:24.601898 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:25.626797 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:25.626761 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:25.730745 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:25.730708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:27.128198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:27.128159 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:27.241964 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:27.241913 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:28.953927 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:28.953885 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c4qqk" Jul 22 01:38:28.990261 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:28.989731 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rh9g2" event={"ID":"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18","Type":"ContainerStarted","Data":"9850904d58a9b670836678a99389321ab9dab75852d01d57ee572e9b5ad40b70"} Jul 22 01:38:28.994014 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:28.993168 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" event={"ID":"e034cd3d-685e-4594-85e8-413cb2b330a1","Type":"ContainerStarted","Data":"d095a56a85769a7b5976dae7ba46996bd1d4baebbedd3f499dee7aad659d6125"} Jul 22 01:38:28.994014 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:28.993233 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" event={"ID":"e034cd3d-685e-4594-85e8-413cb2b330a1","Type":"ContainerStarted","Data":"966cb1607598f5766579800981295001a8e17ba219d397e419c3b827d416baa8"} Jul 22 01:38:28.994014 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:28.993255 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" event={"ID":"e034cd3d-685e-4594-85e8-413cb2b330a1","Type":"ContainerStarted","Data":"250240f6145c346a59ab9bae629885415f399272381399c7c8ed257a2ff3a7a5"} Jul 22 01:38:28.996450 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:28.996267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" event={"ID":"6af2f41c-898d-48ab-b7b2-ca2214f2d0f7","Type":"ContainerStarted","Data":"1777f026f864675b432021056042b32c8cbd5a00499116ca6ee9389b98fc8031"} Jul 22 01:38:28.996647 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:28.996609 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:28.998173 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:28.998134 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" Jul 22 01:38:29.026074 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:29.025961 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76768768fb-c6ztn" podStartSLOduration=2.2217629580000002 podStartE2EDuration="16.025940568s" totalImagesPullingTime="13.80417761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:29.021626707 +0000 UTC m=+83.519068065" watchObservedRunningTime="2026-07-22 01:38:29.025940568 +0000 UTC m=+83.523381926" Jul 22 01:38:29.049604 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:29.049116 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8554748f-d4ds7" podStartSLOduration=2.211503704 podStartE2EDuration="16.049098351s" totalImagesPullingTime="13.837594647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:29.04814747 +0000 UTC m=+83.545588828" watchObservedRunningTime="2026-07-22 01:38:29.049098351 +0000 UTC m=+83.546539708" Jul 22 01:38:29.219996 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:29.219911 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:29.342433 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:29.342384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:31.343121 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:31.343081 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:31.444196 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:31.444149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:31.706983 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:31.706942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:32.009445 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:32.009370 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rh9g2" event={"ID":"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18","Type":"ContainerStarted","Data":"2297a63391e881f0879df091c76bc5741ebc860062cfe3b4035e6159c5afb90a"} Jul 22 01:38:33.558445 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:33.558396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:34.037787 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.037712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:34.162969 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.162926 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c97kn"] Jul 22 01:38:34.166315 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.166287 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.168781 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.168552 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 01:38:34.168911 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.168809 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 01:38:34.169107 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.168979 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 01:38:34.170054 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.169531 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 01:38:34.170054 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.169718 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 01:38:34.170054 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.169894 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-npxl9\"" Jul 22 01:38:34.170054 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.169895 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 01:38:34.171584 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.171536 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:34.289137 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-textfile\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.289137 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289102 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-root\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.289137 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-sys\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.289391 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.289391 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289224 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-wtmp\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.289391 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.289391 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289292 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-metrics-client-ca\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.289391 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289314 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwt5r\" (UniqueName: \"kubernetes.io/projected/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-kube-api-access-rwt5r\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.289391 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.289359 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-tls\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.335076 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.335038 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:34.390228 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-wtmp\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390228 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390189 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-metrics-client-ca\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwt5r\" (UniqueName: \"kubernetes.io/projected/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-kube-api-access-rwt5r\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390288 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-tls\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390312 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-wtmp\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390323 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-textfile\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-root\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390657 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-sys\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390657 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390471 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390657 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390617 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-textfile\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.390922 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.390906 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-root\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.391105 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:38:34.391051 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 01:38:34.391105 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.391063 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-sys\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.391105 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.391085 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.391105 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:38:34.391124 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-tls podName:cdd794b8-ad2e-44a5-a88f-e25a2fb432a1 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:34.891104981 +0000 UTC m=+89.388546321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-tls") pod "node-exporter-c97kn" (UID: "cdd794b8-ad2e-44a5-a88f-e25a2fb432a1") : secret "node-exporter-tls" not found Jul 22 01:38:34.391655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.391632 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-metrics-client-ca\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.393263 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.393241 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.406613 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.406583 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwt5r\" (UniqueName: \"kubernetes.io/projected/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-kube-api-access-rwt5r\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.855820 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.855785 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:34.894775 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.894738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-tls\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.897689 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.897668 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cdd794b8-ad2e-44a5-a88f-e25a2fb432a1-node-exporter-tls\") pod \"node-exporter-c97kn\" (UID: \"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1\") " pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:34.956807 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:34.956759 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:35.071642 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:35.071604 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:35.078205 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:35.078169 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c97kn" Jul 22 01:38:35.163642 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:35.163564 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:35.849842 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:35.849807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:36.116705 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:36.116625 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:36.196375 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:38:36.196344 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd794b8_ad2e_44a5_a88f_e25a2fb432a1.slice/crio-6cd51da8b45c42584d5377d5f852cecf042b8b3a2f1177dcc3006887b685451f WatchSource:0}: Error finding container 6cd51da8b45c42584d5377d5f852cecf042b8b3a2f1177dcc3006887b685451f: Status 404 returned error can't find the container with id 6cd51da8b45c42584d5377d5f852cecf042b8b3a2f1177dcc3006887b685451f Jul 22 01:38:36.221540 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:36.221511 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:36.728706 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:36.728670 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:36.832281 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:36.832249 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:36.926025 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:36.925988 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:37.029276 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:37.029180 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c97kn" event={"ID":"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1","Type":"ContainerStarted","Data":"6cd51da8b45c42584d5377d5f852cecf042b8b3a2f1177dcc3006887b685451f"} Jul 22 01:38:37.031550 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:37.031521 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rh9g2" event={"ID":"c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18","Type":"ContainerStarted","Data":"ba0c2b0e2101a83fecc54c052e3a6025fc87b8b5fae740b4d1c86ad5a2c1cb4c"} Jul 22 01:38:37.055391 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:37.055344 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rh9g2" podStartSLOduration=33.932984797 podStartE2EDuration="49.055325063s" totalImagesPullingTime="15.122340266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:37.054789173 +0000 UTC m=+91.552230532" watchObservedRunningTime="2026-07-22 01:38:37.055325063 +0000 UTC m=+91.552766419" Jul 22 01:38:37.112998 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:37.112928 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:37.550450 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:37.550365 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:38.036225 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:38.036193 2570 generic.go:356] "Generic (PLEG): container finished" podID="cdd794b8-ad2e-44a5-a88f-e25a2fb432a1" containerID="b6afb1fd03c8eee1e30fe4fb966143b493c4be851f44c387fbc9e9308e916434" exitCode=0 Jul 22 01:38:38.036415 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:38.036278 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c97kn" event={"ID":"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1","Type":"ContainerDied","Data":"b6afb1fd03c8eee1e30fe4fb966143b493c4be851f44c387fbc9e9308e916434"} Jul 22 01:38:38.587502 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:38.587467 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:38.654808 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:38.654769 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:38.769436 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:38.769378 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:38.870687 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:38.870653 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:38.952959 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:38.952928 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:39.041753 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:39.041714 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c97kn" event={"ID":"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1","Type":"ContainerStarted","Data":"1843c727da52504d18f25e54f9ec130c0e36e09046e8af5f614b6876dad92b03"} Jul 22 01:38:39.041888 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:39.041769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c97kn" event={"ID":"cdd794b8-ad2e-44a5-a88f-e25a2fb432a1","Type":"ContainerStarted","Data":"b261ee3eb3bb21b4f021266477e4d29ccf6ff577a437fe815bf524718e55da66"} Jul 22 01:38:39.063382 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:39.063334 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c97kn" podStartSLOduration=3.969242415 podStartE2EDuration="5.063319025s" totalImagesPullingTime="1.029180878s" totalInitContainerRuntime="64.895732ms" isStatefulPod=true podCreationTimestamp="2026-07-22 01:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:39.061373231 +0000 UTC m=+93.558814588" watchObservedRunningTime="2026-07-22 01:38:39.063319025 +0000 UTC m=+93.560760381" Jul 22 01:38:40.144588 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:40.144542 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:40.310852 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:40.310815 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:40.409449 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:40.409358 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:41.325681 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:41.325648 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:41.390594 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:41.390566 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:44.226692 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:44.226654 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:44.326171 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:44.326133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:49.600463 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:49.600428 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:49.701306 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:49.701278 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:49.802157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:49.802117 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:38:49.952828 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:38:49.952745 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rpm7t" Jul 22 01:39:06.108874 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:06.108835 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:06.172884 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:06.172846 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:08.564099 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:08.564062 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:08.630268 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:08.630210 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:08.694261 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:08.694202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:08.992611 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:08.992573 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mkp5b_2ed4451e-6eb6-43e2-b514-daae87017ad2/serve-healthcheck-canary/0.log" Jul 22 01:39:13.171998 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:13.171964 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:13.272547 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:13.272509 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:13.823675 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:13.823636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:13.891208 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:13.891166 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:53.152319 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:53.152277 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:53.280268 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:53.280208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:39:59.042925 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:39:59.042834 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:02.600172 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:02.600138 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:02.715636 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:02.715604 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:02.770990 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:02.770956 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:03.316388 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:03.316348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:03.374736 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:03.374705 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:08.417707 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:08.417677 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:08.476681 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:08.476639 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:20.320076 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:20.320039 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:20.380288 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:20.380238 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:24.970663 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:24.970624 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:25.029316 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:25.029282 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:25.123076 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:25.123028 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:36.142605 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:36.142569 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:36.235635 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:36.235597 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:39.118820 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:39.118787 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:40:39.217877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:40:39.217841 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:15.121203 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:15.121160 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:15.227009 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:15.226969 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:41.772643 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:41.772559 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:41.863711 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:41.863672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:43.188651 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.188619 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:43.770076 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.770044 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb"] Jul 22 01:41:43.772938 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.772922 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" Jul 22 01:41:43.775132 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.775111 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 01:41:43.775280 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.775180 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mfk6m\"" Jul 22 01:41:43.775280 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.775189 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:41:43.786420 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.786395 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb"] Jul 22 01:41:43.863700 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.863665 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8501deb1-5799-4a18-bcf1-e832804a486f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x9gbb\" (UID: \"8501deb1-5799-4a18-bcf1-e832804a486f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" Jul 22 01:41:43.863847 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.863715 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkw4w\" (UniqueName: \"kubernetes.io/projected/8501deb1-5799-4a18-bcf1-e832804a486f-kube-api-access-mkw4w\") pod \"cert-manager-operator-controller-manager-55c444bf86-x9gbb\" (UID: \"8501deb1-5799-4a18-bcf1-e832804a486f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" Jul 22 01:41:43.964418 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.964374 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8501deb1-5799-4a18-bcf1-e832804a486f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x9gbb\" (UID: \"8501deb1-5799-4a18-bcf1-e832804a486f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" Jul 22 01:41:43.964418 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.964426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkw4w\" (UniqueName: \"kubernetes.io/projected/8501deb1-5799-4a18-bcf1-e832804a486f-kube-api-access-mkw4w\") pod \"cert-manager-operator-controller-manager-55c444bf86-x9gbb\" (UID: \"8501deb1-5799-4a18-bcf1-e832804a486f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" Jul 22 01:41:43.964766 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.964741 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8501deb1-5799-4a18-bcf1-e832804a486f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x9gbb\" (UID: \"8501deb1-5799-4a18-bcf1-e832804a486f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" Jul 22 01:41:43.973017 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:43.972987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkw4w\" (UniqueName: \"kubernetes.io/projected/8501deb1-5799-4a18-bcf1-e832804a486f-kube-api-access-mkw4w\") pod \"cert-manager-operator-controller-manager-55c444bf86-x9gbb\" (UID: \"8501deb1-5799-4a18-bcf1-e832804a486f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" Jul 22 01:41:44.081400 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:44.081311 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" Jul 22 01:41:44.207710 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:44.207679 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb"] Jul 22 01:41:44.211200 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:41:44.211170 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8501deb1_5799_4a18_bcf1_e832804a486f.slice/crio-fde5f680994e645c11bcefbce924af859a1f1ebcee094b5272b6bd571eaeb2aa WatchSource:0}: Error finding container fde5f680994e645c11bcefbce924af859a1f1ebcee094b5272b6bd571eaeb2aa: Status 404 returned error can't find the container with id fde5f680994e645c11bcefbce924af859a1f1ebcee094b5272b6bd571eaeb2aa Jul 22 01:41:44.631393 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:44.631362 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" event={"ID":"8501deb1-5799-4a18-bcf1-e832804a486f","Type":"ContainerStarted","Data":"fde5f680994e645c11bcefbce924af859a1f1ebcee094b5272b6bd571eaeb2aa"} Jul 22 01:41:45.443371 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:45.443241 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:45.546925 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:45.546890 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:47.643708 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:47.643675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" event={"ID":"8501deb1-5799-4a18-bcf1-e832804a486f","Type":"ContainerStarted","Data":"30a55e963945798cd4c7d1744e6e8f229d3492585b956e3129a36809bc9ae42d"} Jul 22 01:41:47.665267 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:47.665197 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x9gbb" podStartSLOduration=1.969349496 podStartE2EDuration="4.665180842s" totalImagesPullingTime="2.695831346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:41:47.664064482 +0000 UTC m=+282.161505838" watchObservedRunningTime="2026-07-22 01:41:47.665180842 +0000 UTC m=+282.162622198" Jul 22 01:41:47.768520 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:47.768481 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:47.870858 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:47.870828 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:48.078208 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:48.078124 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:48.189712 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:48.189682 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:50.222285 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:50.222252 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:50.616845 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:50.616811 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:52.421773 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:52.421726 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:52.522530 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:52.522502 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:53.126131 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.126101 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n"] Jul 22 01:41:53.129241 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.129208 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" Jul 22 01:41:53.131568 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.131547 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 01:41:53.132322 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.132306 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vwngb\"" Jul 22 01:41:53.132372 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.132332 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 01:41:53.136301 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.136281 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n"] Jul 22 01:41:53.229636 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.229603 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fzd6n\" (UID: \"8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" Jul 22 01:41:53.229790 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.229659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbj9k\" (UniqueName: \"kubernetes.io/projected/8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9-kube-api-access-hbj9k\") pod \"cert-manager-cainjector-559cdd8c87-fzd6n\" (UID: \"8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" Jul 22 01:41:53.329995 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.329958 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbj9k\" (UniqueName: \"kubernetes.io/projected/8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9-kube-api-access-hbj9k\") pod \"cert-manager-cainjector-559cdd8c87-fzd6n\" (UID: \"8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" Jul 22 01:41:53.330147 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.330016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fzd6n\" (UID: \"8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" Jul 22 01:41:53.339710 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.339676 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fzd6n\" (UID: \"8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" Jul 22 01:41:53.339853 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.339832 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbj9k\" (UniqueName: \"kubernetes.io/projected/8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9-kube-api-access-hbj9k\") pod \"cert-manager-cainjector-559cdd8c87-fzd6n\" (UID: \"8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" Jul 22 01:41:53.438345 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.438271 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" Jul 22 01:41:53.567413 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.567369 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n"] Jul 22 01:41:53.569625 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:41:53.569591 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ca7b66f_4d6e_46c5_98b7_43bdf5f5acd9.slice/crio-4fe100289f6390541ee1ddc58bcebf7fd3b5c366142f6ec936eff9a67a346c3c WatchSource:0}: Error finding container 4fe100289f6390541ee1ddc58bcebf7fd3b5c366142f6ec936eff9a67a346c3c: Status 404 returned error can't find the container with id 4fe100289f6390541ee1ddc58bcebf7fd3b5c366142f6ec936eff9a67a346c3c Jul 22 01:41:53.665160 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:53.665127 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" event={"ID":"8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9","Type":"ContainerStarted","Data":"4fe100289f6390541ee1ddc58bcebf7fd3b5c366142f6ec936eff9a67a346c3c"} Jul 22 01:41:54.764550 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:54.764517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:54.865840 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:54.865797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:57.679975 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:57.679942 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" event={"ID":"8ca7b66f-4d6e-46c5-98b7-43bdf5f5acd9","Type":"ContainerStarted","Data":"9fc1578cea5947d1b48a0f57a04634bfda2e04ff7ada2d95de326a4f2c04f3b4"} Jul 22 01:41:57.696802 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:57.696752 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fzd6n" podStartSLOduration=0.941450735 podStartE2EDuration="4.696737397s" totalImagesPullingTime="3.755286662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:41:57.695504103 +0000 UTC m=+292.192945459" watchObservedRunningTime="2026-07-22 01:41:57.696737397 +0000 UTC m=+292.194178753" Jul 22 01:41:58.098618 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:58.098535 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:58.154120 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:58.154086 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:41:58.400080 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:41:58.400048 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:00.326248 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:00.326196 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:00.432282 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:00.432242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:05.789913 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:05.789882 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:42:05.794454 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:05.794389 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:42:05.796840 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:05.796821 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 01:42:07.878507 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:07.878466 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:07.988043 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:07.988011 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:09.882345 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:09.882310 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:24.095607 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.095564 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:24.187151 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.187120 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65486778df-gzszj"] Jul 22 01:42:24.192989 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.192973 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.196893 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.196863 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 01:42:24.197025 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.196876 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jqptx\"" Jul 22 01:42:24.197025 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.196876 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 01:42:24.197025 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.196876 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:42:24.201496 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.201482 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 01:42:24.202265 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.202246 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 01:42:24.204028 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.204002 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65486778df-gzszj"] Jul 22 01:42:24.348584 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.348502 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/24409476-e5b3-40ad-9afe-7fd4d93a653d-manager-config\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.348584 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.348542 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlzk\" (UniqueName: \"kubernetes.io/projected/24409476-e5b3-40ad-9afe-7fd4d93a653d-kube-api-access-2qlzk\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.348584 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.348567 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24409476-e5b3-40ad-9afe-7fd4d93a653d-metrics-cert\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.348800 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.348593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24409476-e5b3-40ad-9afe-7fd4d93a653d-cert\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.449596 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.449558 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlzk\" (UniqueName: \"kubernetes.io/projected/24409476-e5b3-40ad-9afe-7fd4d93a653d-kube-api-access-2qlzk\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.449596 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.449599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24409476-e5b3-40ad-9afe-7fd4d93a653d-metrics-cert\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.449828 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.449620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24409476-e5b3-40ad-9afe-7fd4d93a653d-cert\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.449828 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.449659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/24409476-e5b3-40ad-9afe-7fd4d93a653d-manager-config\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.450332 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.450305 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/24409476-e5b3-40ad-9afe-7fd4d93a653d-manager-config\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.452357 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.452333 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24409476-e5b3-40ad-9afe-7fd4d93a653d-metrics-cert\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.452468 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.452361 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24409476-e5b3-40ad-9afe-7fd4d93a653d-cert\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.459917 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.459887 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlzk\" (UniqueName: \"kubernetes.io/projected/24409476-e5b3-40ad-9afe-7fd4d93a653d-kube-api-access-2qlzk\") pod \"lws-controller-manager-65486778df-gzszj\" (UID: \"24409476-e5b3-40ad-9afe-7fd4d93a653d\") " pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.502005 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.501970 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:24.629199 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.629176 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65486778df-gzszj"] Jul 22 01:42:24.631919 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:42:24.631896 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24409476_e5b3_40ad_9afe_7fd4d93a653d.slice/crio-70846aab9da04327d8649b7b35bc6e9bd23629f26e0baa2230596a1d6db68f14 WatchSource:0}: Error finding container 70846aab9da04327d8649b7b35bc6e9bd23629f26e0baa2230596a1d6db68f14: Status 404 returned error can't find the container with id 70846aab9da04327d8649b7b35bc6e9bd23629f26e0baa2230596a1d6db68f14 Jul 22 01:42:24.633715 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.633698 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:42:24.765816 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:24.765784 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" event={"ID":"24409476-e5b3-40ad-9afe-7fd4d93a653d","Type":"ContainerStarted","Data":"70846aab9da04327d8649b7b35bc6e9bd23629f26e0baa2230596a1d6db68f14"} Jul 22 01:42:25.634109 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:25.634078 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:26.070362 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.070269 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:26.362860 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.362812 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:26.469202 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.469168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:26.528126 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.528085 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf"] Jul 22 01:42:26.534483 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.534462 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.537623 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.537599 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 01:42:26.537770 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.537694 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 01:42:26.537857 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.537832 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 01:42:26.537978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.537869 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6cfrp\"" Jul 22 01:42:26.537978 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.537921 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 01:42:26.546452 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.546428 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf"] Jul 22 01:42:26.658010 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.657931 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:26.665863 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.665834 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-tmp\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.666002 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.665883 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-manifests\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.666002 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.665904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.666090 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.665999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.666090 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.666025 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kq6\" (UniqueName: \"kubernetes.io/projected/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-kube-api-access-m7kq6\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.766843 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.766806 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-tmp\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.767021 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.766877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-manifests\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.767021 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.766903 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.767021 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.766975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.767021 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.766998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kq6\" (UniqueName: \"kubernetes.io/projected/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-kube-api-access-m7kq6\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.767389 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.767332 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-manifests\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.770275 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.770244 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.770402 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.770248 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-tmp\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.770946 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.770921 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.775960 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.775932 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kq6\" (UniqueName: \"kubernetes.io/projected/77612cb4-fe8d-4092-bf08-73dfb2dd58b1-kube-api-access-m7kq6\") pod \"opendatahub-operator-controller-manager-5c578697c6-jqhsf\" (UID: \"77612cb4-fe8d-4092-bf08-73dfb2dd58b1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:26.801740 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.801712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:26.846109 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:26.846077 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:27.148969 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:27.148933 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:27.261611 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:27.261563 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:27.404565 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:27.404403 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:27.477349 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:27.477323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf"] Jul 22 01:42:27.479062 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:42:27.479027 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77612cb4_fe8d_4092_bf08_73dfb2dd58b1.slice/crio-d912c5bbfdf5c17111b6ae9afd1b3d1f17dc9b29d0a363bcf6a79582b304d5ed WatchSource:0}: Error finding container d912c5bbfdf5c17111b6ae9afd1b3d1f17dc9b29d0a363bcf6a79582b304d5ed: Status 404 returned error can't find the container with id d912c5bbfdf5c17111b6ae9afd1b3d1f17dc9b29d0a363bcf6a79582b304d5ed Jul 22 01:42:27.779477 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:27.779383 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" event={"ID":"24409476-e5b3-40ad-9afe-7fd4d93a653d","Type":"ContainerStarted","Data":"f67689c560a6103723721c4e53588a0e604139e4046a5241d67b5c7a4d5b1c73"} Jul 22 01:42:27.779857 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:27.779481 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:27.780642 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:27.780619 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" event={"ID":"77612cb4-fe8d-4092-bf08-73dfb2dd58b1","Type":"ContainerStarted","Data":"d912c5bbfdf5c17111b6ae9afd1b3d1f17dc9b29d0a363bcf6a79582b304d5ed"} Jul 22 01:42:27.801598 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:27.801546 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" podStartSLOduration=1.029296677 podStartE2EDuration="3.801533554s" totalImagesPullingTime="2.772236877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:27.801308883 +0000 UTC m=+322.298750269" watchObservedRunningTime="2026-07-22 01:42:27.801533554 +0000 UTC m=+322.298974909" Jul 22 01:42:30.796958 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:30.796920 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" event={"ID":"77612cb4-fe8d-4092-bf08-73dfb2dd58b1","Type":"ContainerStarted","Data":"ba3fa5a123ae32cbe384b3e0b001883d4ff93a4480ecee8281c30af887584057"} Jul 22 01:42:31.802186 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:31.802149 2570 generic.go:356] "Generic (PLEG): container finished" podID="77612cb4-fe8d-4092-bf08-73dfb2dd58b1" containerID="ba3fa5a123ae32cbe384b3e0b001883d4ff93a4480ecee8281c30af887584057" exitCode=0 Jul 22 01:42:31.802567 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:31.802202 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" event={"ID":"77612cb4-fe8d-4092-bf08-73dfb2dd58b1","Type":"ContainerDied","Data":"ba3fa5a123ae32cbe384b3e0b001883d4ff93a4480ecee8281c30af887584057"} Jul 22 01:42:32.807325 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:32.807290 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" event={"ID":"77612cb4-fe8d-4092-bf08-73dfb2dd58b1","Type":"ContainerStarted","Data":"0715759e7f267429d5022c20b384051a481f308772f9100ae451c5ab78053abd"} Jul 22 01:42:32.807703 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:32.807411 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:32.832082 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:32.832029 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" podStartSLOduration=2.334093541 podStartE2EDuration="6.832012923s" totalImagesPullingTime="3.481550494s" totalInitContainerRuntime="1.016368888s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:32.830082914 +0000 UTC m=+327.327524296" watchObservedRunningTime="2026-07-22 01:42:32.832012923 +0000 UTC m=+327.329454282" Jul 22 01:42:38.788074 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:38.788042 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65486778df-gzszj" Jul 22 01:42:43.813527 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:43.813489 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-jqhsf" Jul 22 01:42:45.937488 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:45.937450 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:46.077758 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.077721 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:46.373692 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.373660 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:46.500378 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.500340 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r"] Jul 22 01:42:46.509846 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.509817 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.512412 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.512386 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 01:42:46.513341 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.513321 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 01:42:46.513463 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.513374 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 01:42:46.513463 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.513455 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h6x54\"" Jul 22 01:42:46.513748 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.513730 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 01:42:46.525231 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.523137 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r"] Jul 22 01:42:46.612751 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.612714 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-tls-certs\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.612751 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.612752 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xgp\" (UniqueName: \"kubernetes.io/projected/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-kube-api-access-p9xgp\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.612999 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.612784 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-tmp\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.644454 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.644363 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:46.713840 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.713800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-tmp\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.714022 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.713890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-tls-certs\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.714022 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.713916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xgp\" (UniqueName: \"kubernetes.io/projected/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-kube-api-access-p9xgp\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.716649 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.716615 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-tmp\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.717569 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.717544 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-tls-certs\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.735547 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.735519 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xgp\" (UniqueName: \"kubernetes.io/projected/65e18150-c9ce-46ff-9d14-8f5dc63c5de4-kube-api-access-p9xgp\") pod \"kube-auth-proxy-5975cd7b94-vhl8r\" (UID: \"65e18150-c9ce-46ff-9d14-8f5dc63c5de4\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.745606 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.745579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:46.823200 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.823173 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" Jul 22 01:42:46.880743 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.880705 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:46.963865 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.963837 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r"] Jul 22 01:42:46.966669 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:42:46.966640 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e18150_c9ce_46ff_9d14_8f5dc63c5de4.slice/crio-036dee37fdea20dc2255a056d58cfb874cb4f5c5e215e3c6ece73b2b5606e8c9 WatchSource:0}: Error finding container 036dee37fdea20dc2255a056d58cfb874cb4f5c5e215e3c6ece73b2b5606e8c9: Status 404 returned error can't find the container with id 036dee37fdea20dc2255a056d58cfb874cb4f5c5e215e3c6ece73b2b5606e8c9 Jul 22 01:42:46.987710 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:46.987675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:47.476248 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:47.476201 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:47.773325 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:47.773200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:47.864564 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:47.864491 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" event={"ID":"65e18150-c9ce-46ff-9d14-8f5dc63c5de4","Type":"ContainerStarted","Data":"036dee37fdea20dc2255a056d58cfb874cb4f5c5e215e3c6ece73b2b5606e8c9"} Jul 22 01:42:48.078499 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:48.078422 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:48.175018 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:48.174989 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:49.303995 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:49.303955 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:49.873461 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:49.873415 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" event={"ID":"65e18150-c9ce-46ff-9d14-8f5dc63c5de4","Type":"ContainerStarted","Data":"6891ce2650f7e524eef472b135c6c459e3a36cf0d3535b96d58d9f5771b97177"} Jul 22 01:42:49.893497 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:49.893448 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-vhl8r" podStartSLOduration=1.166248231 podStartE2EDuration="3.893434574s" totalImagesPullingTime="2.727186343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:49.892295171 +0000 UTC m=+344.389736548" watchObservedRunningTime="2026-07-22 01:42:49.893434574 +0000 UTC m=+344.390875931" Jul 22 01:42:53.702864 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:53.702830 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:53.765158 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:53.765124 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:54.009889 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:54.009809 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:54.142108 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:54.142073 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:54.243570 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:54.243532 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:57.733203 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:57.733168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:57.821970 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:57.821939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:58.127064 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:58.127024 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:58.378493 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:58.378410 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:58.980736 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:58.980704 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:59.089021 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:59.088987 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:42:59.194116 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:42:59.194072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:43:47.513579 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:43:47.513535 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:43:47.580203 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:43:47.580170 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:43:47.709068 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:43:47.709029 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:43:54.194313 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:43:54.194264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:16.456500 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:16.456458 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:20.061644 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:20.061611 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:20.164667 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:20.164627 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:24.662796 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:24.662756 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:25.064682 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:25.064602 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:26.511636 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:26.511599 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:28.068095 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:28.067978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:28.940065 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:28.940031 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:30.911200 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:30.911168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:32.274566 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:32.274534 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:32.374589 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:32.374554 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:33.747176 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.747138 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5"] Jul 22 01:44:33.750379 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.750357 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.753267 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.753229 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 22 01:44:33.753385 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.753370 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2f55z\"" Jul 22 01:44:33.754014 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.753990 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 01:44:33.754115 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.753994 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 22 01:44:33.754115 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.754054 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 01:44:33.755808 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.755786 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:33.765461 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.765438 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5"] Jul 22 01:44:33.795470 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.795435 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpj5g\" (UniqueName: \"kubernetes.io/projected/52d390f5-a778-4027-8413-3162d947a38d-kube-api-access-wpj5g\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.795603 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.795494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/52d390f5-a778-4027-8413-3162d947a38d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.795603 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.795557 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/52d390f5-a778-4027-8413-3162d947a38d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.896515 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.896475 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpj5g\" (UniqueName: \"kubernetes.io/projected/52d390f5-a778-4027-8413-3162d947a38d-kube-api-access-wpj5g\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.897444 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.896531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/52d390f5-a778-4027-8413-3162d947a38d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.897444 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.896950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/52d390f5-a778-4027-8413-3162d947a38d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.898007 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.897984 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/52d390f5-a778-4027-8413-3162d947a38d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.902468 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.902316 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/52d390f5-a778-4027-8413-3162d947a38d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.914898 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.914869 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpj5g\" (UniqueName: \"kubernetes.io/projected/52d390f5-a778-4027-8413-3162d947a38d-kube-api-access-wpj5g\") pod \"kuadrant-console-plugin-66b5774985-tnwm5\" (UID: \"52d390f5-a778-4027-8413-3162d947a38d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:33.941367 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:33.941339 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:34.061094 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:34.061019 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" Jul 22 01:44:34.185109 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:34.185044 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5"] Jul 22 01:44:34.188069 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:44:34.188035 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52d390f5_a778_4027_8413_3162d947a38d.slice/crio-9200d53b0b23b888f662351a04dd5bf88e168a29221067586d4f7d761fb6d8dc WatchSource:0}: Error finding container 9200d53b0b23b888f662351a04dd5bf88e168a29221067586d4f7d761fb6d8dc: Status 404 returned error can't find the container with id 9200d53b0b23b888f662351a04dd5bf88e168a29221067586d4f7d761fb6d8dc Jul 22 01:44:34.256497 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:34.256462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" event={"ID":"52d390f5-a778-4027-8413-3162d947a38d","Type":"ContainerStarted","Data":"9200d53b0b23b888f662351a04dd5bf88e168a29221067586d4f7d761fb6d8dc"} Jul 22 01:44:35.671956 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:35.671919 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:35.737881 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:35.737837 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:44.301941 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:44.301879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:44.403099 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:44.403017 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:46.755633 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:46.755598 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:46.850414 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:46.850354 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:51.119841 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:51.119803 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:51.213018 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:51.212983 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:53.282046 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:53.282003 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:53.337837 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:53.337794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:44:57.361817 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:57.361781 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" event={"ID":"52d390f5-a778-4027-8413-3162d947a38d","Type":"ContainerStarted","Data":"36a1e1c7e6dc110f99e259bd37512cf4e1b0d61c1e00b19476511e6180aa53a6"} Jul 22 01:44:57.378084 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:44:57.378030 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tnwm5" podStartSLOduration=1.9671327440000002 podStartE2EDuration="24.378014612s" totalImagesPullingTime="22.410881868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:57.376391392 +0000 UTC m=+471.873832749" watchObservedRunningTime="2026-07-22 01:44:57.378014612 +0000 UTC m=+471.875455969" Jul 22 01:45:02.044536 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:02.044499 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:02.145431 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:02.145396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:04.489554 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:04.489522 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:04.549921 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:04.549891 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:06.403060 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:06.403024 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:08.874431 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:08.874395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:08.977157 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:08.977122 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:11.852921 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:11.852879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:11.922325 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:11.922293 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:12.667330 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:12.667295 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:13.093882 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:13.093803 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:17.214611 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:17.214576 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:17.314533 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:17.314501 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:17.416396 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:17.416361 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:21.064725 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:21.064696 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:21.162879 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:21.162849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:23.511916 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:23.511882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:23.570542 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:23.570502 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:27.207483 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.207444 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:27.316716 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.316675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:27.415299 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.415256 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:27.561901 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.561823 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:45:27.696280 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.696245 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:45:27.696462 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.696290 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:45:27.696462 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.696342 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:27.699433 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.699409 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 01:45:27.734897 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.734872 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f983e51-63a3-4406-bef9-9acb80a726c1-config-file\") pod \"limitador-limitador-54d6d988d5-kwdcn\" (UID: \"5f983e51-63a3-4406-bef9-9acb80a726c1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:27.735038 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.734924 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6x2\" (UniqueName: \"kubernetes.io/projected/5f983e51-63a3-4406-bef9-9acb80a726c1-kube-api-access-9z6x2\") pod \"limitador-limitador-54d6d988d5-kwdcn\" (UID: \"5f983e51-63a3-4406-bef9-9acb80a726c1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:27.835463 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.835384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f983e51-63a3-4406-bef9-9acb80a726c1-config-file\") pod \"limitador-limitador-54d6d988d5-kwdcn\" (UID: \"5f983e51-63a3-4406-bef9-9acb80a726c1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:27.835463 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.835441 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6x2\" (UniqueName: \"kubernetes.io/projected/5f983e51-63a3-4406-bef9-9acb80a726c1-kube-api-access-9z6x2\") pod \"limitador-limitador-54d6d988d5-kwdcn\" (UID: \"5f983e51-63a3-4406-bef9-9acb80a726c1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:27.836109 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.836087 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5f983e51-63a3-4406-bef9-9acb80a726c1-config-file\") pod \"limitador-limitador-54d6d988d5-kwdcn\" (UID: \"5f983e51-63a3-4406-bef9-9acb80a726c1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:27.844683 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:27.844656 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6x2\" (UniqueName: \"kubernetes.io/projected/5f983e51-63a3-4406-bef9-9acb80a726c1-kube-api-access-9z6x2\") pod \"limitador-limitador-54d6d988d5-kwdcn\" (UID: \"5f983e51-63a3-4406-bef9-9acb80a726c1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:28.006344 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.006307 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:28.076764 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.076729 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-n4nc8"] Jul 22 01:45:28.094133 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.094108 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-n4nc8"] Jul 22 01:45:28.094304 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.094201 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n4nc8" Jul 22 01:45:28.096466 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.096444 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7qvb4\"" Jul 22 01:45:28.112579 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.112552 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:28.138498 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.138461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgr5v\" (UniqueName: \"kubernetes.io/projected/8e933e49-3457-4260-b98d-8d7979cfc832-kube-api-access-mgr5v\") pod \"authorino-58cd955675-n4nc8\" (UID: \"8e933e49-3457-4260-b98d-8d7979cfc832\") " pod="kuadrant-system/authorino-58cd955675-n4nc8" Jul 22 01:45:28.175230 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.175190 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:28.213156 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.212970 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:45:28.215495 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:45:28.215465 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f983e51_63a3_4406_bef9_9acb80a726c1.slice/crio-c8b5f868a17fa38f5009e21e63a880600473d516144091d03021b8554ff9c357 WatchSource:0}: Error finding container c8b5f868a17fa38f5009e21e63a880600473d516144091d03021b8554ff9c357: Status 404 returned error can't find the container with id c8b5f868a17fa38f5009e21e63a880600473d516144091d03021b8554ff9c357 Jul 22 01:45:28.239498 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.239474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgr5v\" (UniqueName: \"kubernetes.io/projected/8e933e49-3457-4260-b98d-8d7979cfc832-kube-api-access-mgr5v\") pod \"authorino-58cd955675-n4nc8\" (UID: \"8e933e49-3457-4260-b98d-8d7979cfc832\") " pod="kuadrant-system/authorino-58cd955675-n4nc8" Jul 22 01:45:28.249554 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.249531 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgr5v\" (UniqueName: \"kubernetes.io/projected/8e933e49-3457-4260-b98d-8d7979cfc832-kube-api-access-mgr5v\") pod \"authorino-58cd955675-n4nc8\" (UID: \"8e933e49-3457-4260-b98d-8d7979cfc832\") " pod="kuadrant-system/authorino-58cd955675-n4nc8" Jul 22 01:45:28.413246 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.413196 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n4nc8" Jul 22 01:45:28.477299 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.477260 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" event={"ID":"5f983e51-63a3-4406-bef9-9acb80a726c1","Type":"ContainerStarted","Data":"c8b5f868a17fa38f5009e21e63a880600473d516144091d03021b8554ff9c357"} Jul 22 01:45:28.534719 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:28.534685 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-n4nc8"] Jul 22 01:45:28.537475 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:45:28.537437 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e933e49_3457_4260_b98d_8d7979cfc832.slice/crio-e4f613aa5a95fad1d171cf6f79115b4a8feec25c1cbbdcd783e752716bd8148b WatchSource:0}: Error finding container e4f613aa5a95fad1d171cf6f79115b4a8feec25c1cbbdcd783e752716bd8148b: Status 404 returned error can't find the container with id e4f613aa5a95fad1d171cf6f79115b4a8feec25c1cbbdcd783e752716bd8148b Jul 22 01:45:29.486877 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:29.486833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n4nc8" event={"ID":"8e933e49-3457-4260-b98d-8d7979cfc832","Type":"ContainerStarted","Data":"e4f613aa5a95fad1d171cf6f79115b4a8feec25c1cbbdcd783e752716bd8148b"} Jul 22 01:45:32.484691 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:32.483474 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-n4nc8"] Jul 22 01:45:33.507289 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.507256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" event={"ID":"5f983e51-63a3-4406-bef9-9acb80a726c1","Type":"ContainerStarted","Data":"b24f6e4462575d153a10815fe423a804d3a92119ccc7d9d80a3ebe36277d10db"} Jul 22 01:45:33.507803 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.507358 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:33.508942 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.508923 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n4nc8" event={"ID":"8e933e49-3457-4260-b98d-8d7979cfc832","Type":"ContainerStarted","Data":"d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295"} Jul 22 01:45:33.509019 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.509000 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-n4nc8" podUID="8e933e49-3457-4260-b98d-8d7979cfc832" containerName="authorino" containerID="cri-o://d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295" gracePeriod=30 Jul 22 01:45:33.530776 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.530735 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" podStartSLOduration=2.236814918 podStartE2EDuration="6.530720471s" totalImagesPullingTime="4.293905553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:33.530383247 +0000 UTC m=+508.027824620" watchObservedRunningTime="2026-07-22 01:45:33.530720471 +0000 UTC m=+508.028161828" Jul 22 01:45:33.550891 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.550844 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-n4nc8" podStartSLOduration=1.5687367349999999 podStartE2EDuration="5.550831268s" totalImagesPullingTime="3.982094533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:33.550105496 +0000 UTC m=+508.047546869" watchObservedRunningTime="2026-07-22 01:45:33.550831268 +0000 UTC m=+508.048272624" Jul 22 01:45:33.757390 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.757327 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n4nc8" Jul 22 01:45:33.790136 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.790106 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgr5v\" (UniqueName: \"kubernetes.io/projected/8e933e49-3457-4260-b98d-8d7979cfc832-kube-api-access-mgr5v\") pod \"8e933e49-3457-4260-b98d-8d7979cfc832\" (UID: \"8e933e49-3457-4260-b98d-8d7979cfc832\") " Jul 22 01:45:33.792194 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.792171 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e933e49-3457-4260-b98d-8d7979cfc832-kube-api-access-mgr5v" (OuterVolumeSpecName: "kube-api-access-mgr5v") pod "8e933e49-3457-4260-b98d-8d7979cfc832" (UID: "8e933e49-3457-4260-b98d-8d7979cfc832"). InnerVolumeSpecName "kube-api-access-mgr5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:45:33.891348 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:33.891314 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgr5v\" (UniqueName: \"kubernetes.io/projected/8e933e49-3457-4260-b98d-8d7979cfc832-kube-api-access-mgr5v\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:45:34.514967 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.514930 2570 generic.go:356] "Generic (PLEG): container finished" podID="8e933e49-3457-4260-b98d-8d7979cfc832" containerID="d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295" exitCode=0 Jul 22 01:45:34.514967 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.514970 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n4nc8" Jul 22 01:45:34.515476 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.515020 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n4nc8" event={"ID":"8e933e49-3457-4260-b98d-8d7979cfc832","Type":"ContainerDied","Data":"d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295"} Jul 22 01:45:34.515476 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.515061 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n4nc8" event={"ID":"8e933e49-3457-4260-b98d-8d7979cfc832","Type":"ContainerDied","Data":"e4f613aa5a95fad1d171cf6f79115b4a8feec25c1cbbdcd783e752716bd8148b"} Jul 22 01:45:34.515476 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.515088 2570 scope.go:117] "RemoveContainer" containerID="d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295" Jul 22 01:45:34.523671 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.523654 2570 scope.go:117] "RemoveContainer" containerID="d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295" Jul 22 01:45:34.523908 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:45:34.523889 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295\": container with ID starting with d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295 not found: ID does not exist" containerID="d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295" Jul 22 01:45:34.523979 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.523920 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295"} err="failed to get container status \"d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295\": rpc error: code = NotFound desc = could not find container \"d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295\": container with ID starting with d1a71c9f822a00bcdf42ca31e245414489509f661a7ddda21980571d87563295 not found: ID does not exist" Jul 22 01:45:34.535819 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.535792 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-n4nc8"] Jul 22 01:45:34.537816 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:34.537795 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-n4nc8"] Jul 22 01:45:35.597810 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:35.597783 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e933e49-3457-4260-b98d-8d7979cfc832" path="/var/lib/kubelet/pods/8e933e49-3457-4260-b98d-8d7979cfc832/volumes" Jul 22 01:45:44.516134 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:44.516105 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-kwdcn" Jul 22 01:45:46.659089 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:46.659049 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:46.754743 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:46.754706 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:47.229339 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:45:47.229302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:00.941852 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:00.941770 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.213634 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.213555 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8jjth"] Jul 22 01:46:01.213850 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.213837 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e933e49-3457-4260-b98d-8d7979cfc832" containerName="authorino" Jul 22 01:46:01.213894 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.213852 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e933e49-3457-4260-b98d-8d7979cfc832" containerName="authorino" Jul 22 01:46:01.213924 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.213910 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e933e49-3457-4260-b98d-8d7979cfc832" containerName="authorino" Jul 22 01:46:01.216767 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.216748 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8jjth" Jul 22 01:46:01.218993 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.218969 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7qvb4\"" Jul 22 01:46:01.223326 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.223305 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8jjth"] Jul 22 01:46:01.244616 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.244584 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.300342 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.300300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kblv9\" (UniqueName: \"kubernetes.io/projected/f6911f2c-165e-4b3b-a091-c2c6d42815e7-kube-api-access-kblv9\") pod \"authorino-5d4b67c446-8jjth\" (UID: \"f6911f2c-165e-4b3b-a091-c2c6d42815e7\") " pod="kuadrant-system/authorino-5d4b67c446-8jjth" Jul 22 01:46:01.401166 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.401135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kblv9\" (UniqueName: \"kubernetes.io/projected/f6911f2c-165e-4b3b-a091-c2c6d42815e7-kube-api-access-kblv9\") pod \"authorino-5d4b67c446-8jjth\" (UID: \"f6911f2c-165e-4b3b-a091-c2c6d42815e7\") " pod="kuadrant-system/authorino-5d4b67c446-8jjth" Jul 22 01:46:01.410026 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.409998 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kblv9\" (UniqueName: \"kubernetes.io/projected/f6911f2c-165e-4b3b-a091-c2c6d42815e7-kube-api-access-kblv9\") pod \"authorino-5d4b67c446-8jjth\" (UID: \"f6911f2c-165e-4b3b-a091-c2c6d42815e7\") " pod="kuadrant-system/authorino-5d4b67c446-8jjth" Jul 22 01:46:01.484159 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.484068 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8jjth"] Jul 22 01:46:01.484359 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.484337 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8jjth" Jul 22 01:46:01.609182 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.609151 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8jjth"] Jul 22 01:46:01.611981 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:46:01.611950 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6911f2c_165e_4b3b_a091_c2c6d42815e7.slice/crio-abbc5b5875227dfbea06e4afd25ebf397de42ec77ffeeb3be2865072dcc6c112 WatchSource:0}: Error finding container abbc5b5875227dfbea06e4afd25ebf397de42ec77ffeeb3be2865072dcc6c112: Status 404 returned error can't find the container with id abbc5b5875227dfbea06e4afd25ebf397de42ec77ffeeb3be2865072dcc6c112 Jul 22 01:46:01.619399 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:01.619376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-8jjth" event={"ID":"f6911f2c-165e-4b3b-a091-c2c6d42815e7","Type":"ContainerStarted","Data":"abbc5b5875227dfbea06e4afd25ebf397de42ec77ffeeb3be2865072dcc6c112"} Jul 22 01:46:02.624093 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:02.624054 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-8jjth" event={"ID":"f6911f2c-165e-4b3b-a091-c2c6d42815e7","Type":"ContainerStarted","Data":"3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea"} Jul 22 01:46:02.624093 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:02.624068 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-8jjth" podUID="f6911f2c-165e-4b3b-a091-c2c6d42815e7" containerName="authorino" containerID="cri-o://3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea" gracePeriod=30 Jul 22 01:46:02.639655 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:02.639611 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-8jjth" podStartSLOduration=1.233847258 podStartE2EDuration="1.639595966s" totalImagesPullingTime="405.748708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:02.638603605 +0000 UTC m=+537.136044974" watchObservedRunningTime="2026-07-22 01:46:02.639595966 +0000 UTC m=+537.137037322" Jul 22 01:46:02.862975 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:02.862952 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8jjth" Jul 22 01:46:03.015682 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.015591 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kblv9\" (UniqueName: \"kubernetes.io/projected/f6911f2c-165e-4b3b-a091-c2c6d42815e7-kube-api-access-kblv9\") pod \"f6911f2c-165e-4b3b-a091-c2c6d42815e7\" (UID: \"f6911f2c-165e-4b3b-a091-c2c6d42815e7\") " Jul 22 01:46:03.017696 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.017676 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6911f2c-165e-4b3b-a091-c2c6d42815e7-kube-api-access-kblv9" (OuterVolumeSpecName: "kube-api-access-kblv9") pod "f6911f2c-165e-4b3b-a091-c2c6d42815e7" (UID: "f6911f2c-165e-4b3b-a091-c2c6d42815e7"). InnerVolumeSpecName "kube-api-access-kblv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:46:03.116132 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.116104 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kblv9\" (UniqueName: \"kubernetes.io/projected/f6911f2c-165e-4b3b-a091-c2c6d42815e7-kube-api-access-kblv9\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 01:46:03.628792 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.628758 2570 generic.go:356] "Generic (PLEG): container finished" podID="f6911f2c-165e-4b3b-a091-c2c6d42815e7" containerID="3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea" exitCode=0 Jul 22 01:46:03.629141 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.628807 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8jjth" Jul 22 01:46:03.629141 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.628839 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-8jjth" event={"ID":"f6911f2c-165e-4b3b-a091-c2c6d42815e7","Type":"ContainerDied","Data":"3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea"} Jul 22 01:46:03.629141 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.628878 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-8jjth" event={"ID":"f6911f2c-165e-4b3b-a091-c2c6d42815e7","Type":"ContainerDied","Data":"abbc5b5875227dfbea06e4afd25ebf397de42ec77ffeeb3be2865072dcc6c112"} Jul 22 01:46:03.629141 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.628896 2570 scope.go:117] "RemoveContainer" containerID="3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea" Jul 22 01:46:03.636579 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.636553 2570 scope.go:117] "RemoveContainer" containerID="3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea" Jul 22 01:46:03.636804 ip-10-0-130-174 kubenswrapper[2570]: E0722 01:46:03.636786 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea\": container with ID starting with 3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea not found: ID does not exist" containerID="3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea" Jul 22 01:46:03.636867 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.636812 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea"} err="failed to get container status \"3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea\": rpc error: code = NotFound desc = could not find container \"3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea\": container with ID starting with 3db72c139f66eaf08f5a8c770b3df8de4aa6ae4b284a276f7f31ed88faf149ea not found: ID does not exist" Jul 22 01:46:03.645481 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.645462 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8jjth"] Jul 22 01:46:03.650919 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:03.650900 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8jjth"] Jul 22 01:46:05.597729 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:05.597688 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6911f2c-165e-4b3b-a091-c2c6d42815e7" path="/var/lib/kubelet/pods/f6911f2c-165e-4b3b-a091-c2c6d42815e7/volumes" Jul 22 01:46:07.538691 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:07.538654 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:07.634840 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:07.634796 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:07.731030 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:07.730993 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:07.793503 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:07.793430 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:08.993508 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:08.993475 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:09.044442 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:09.044413 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:10.834627 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:10.834585 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:10.941617 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:10.941582 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:42.636551 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:42.636518 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:42.738259 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:42.738206 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:43.080036 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:43.079957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:43.339096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:43.339015 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:43.729530 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:43.729499 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:43.825001 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:43.824963 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:43.922599 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:43.922565 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:46:47.571979 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:47.571951 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:53.021861 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:53.021832 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:53.156416 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:53.156383 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:59.728096 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:46:59.728060 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:00.086620 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:00.086537 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:47:05.812374 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:05.812346 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:47:05.813546 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:05.813518 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:47:09.528364 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:09.528327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:10.413090 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:10.413053 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:47:17.895674 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.895627 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h"] Jul 22 01:47:17.896127 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.895921 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6911f2c-165e-4b3b-a091-c2c6d42815e7" containerName="authorino" Jul 22 01:47:17.896127 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.895932 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6911f2c-165e-4b3b-a091-c2c6d42815e7" containerName="authorino" Jul 22 01:47:17.896127 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.895987 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6911f2c-165e-4b3b-a091-c2c6d42815e7" containerName="authorino" Jul 22 01:47:17.899014 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.898985 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:17.903091 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.903066 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v8lsh\"" Jul 22 01:47:17.903091 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.903082 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 01:47:17.903310 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.903066 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 01:47:17.903310 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.903075 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 22 01:47:17.912296 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.912276 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h"] Jul 22 01:47:17.945617 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.945576 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:17.947615 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.947596 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9k5f\" (UniqueName: \"kubernetes.io/projected/6f73490b-4058-4566-ac39-ecdce85f4481-kube-api-access-s9k5f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:17.947687 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.947631 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:17.947687 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.947650 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f73490b-4058-4566-ac39-ecdce85f4481-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:17.947687 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.947681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:17.947807 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.947722 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:17.947807 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:17.947738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049067 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9k5f\" (UniqueName: \"kubernetes.io/projected/6f73490b-4058-4566-ac39-ecdce85f4481-kube-api-access-s9k5f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049258 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049258 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049100 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f73490b-4058-4566-ac39-ecdce85f4481-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049383 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049284 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049383 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049354 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049488 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049388 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049531 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049502 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049649 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049628 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.049702 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.049681 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.051592 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.051566 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f73490b-4058-4566-ac39-ecdce85f4481-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.051702 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.051673 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f73490b-4058-4566-ac39-ecdce85f4481-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.059025 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.058998 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9k5f\" (UniqueName: \"kubernetes.io/projected/6f73490b-4058-4566-ac39-ecdce85f4481-kube-api-access-s9k5f\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hn88h\" (UID: \"6f73490b-4058-4566-ac39-ecdce85f4481\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.209954 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.209863 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:18.340174 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.340102 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h"] Jul 22 01:47:18.342665 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:47:18.342632 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f73490b_4058_4566_ac39_ecdce85f4481.slice/crio-61953c0d3861b00497b079c3913c4e4f6efa40105a0698285bc6a82f1eba8cd1 WatchSource:0}: Error finding container 61953c0d3861b00497b079c3913c4e4f6efa40105a0698285bc6a82f1eba8cd1: Status 404 returned error can't find the container with id 61953c0d3861b00497b079c3913c4e4f6efa40105a0698285bc6a82f1eba8cd1 Jul 22 01:47:18.896870 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.896832 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:47:18.907762 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:18.907722 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" event={"ID":"6f73490b-4058-4566-ac39-ecdce85f4481","Type":"ContainerStarted","Data":"61953c0d3861b00497b079c3913c4e4f6efa40105a0698285bc6a82f1eba8cd1"} Jul 22 01:47:19.913804 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:19.913770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" event={"ID":"6f73490b-4058-4566-ac39-ecdce85f4481","Type":"ContainerStarted","Data":"c58908f966b633e2e569785342f119d00cf66246c002c9a12221e8b9d59f27f5"} Jul 22 01:47:19.933290 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:19.933242 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" podStartSLOduration=1.956586795 podStartE2EDuration="2.933208519s" totalImagesPullingTime="976.621724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:19.932261663 +0000 UTC m=+614.429703032" watchObservedRunningTime="2026-07-22 01:47:19.933208519 +0000 UTC m=+614.430649876" Jul 22 01:47:25.027782 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:25.027746 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:26.000271 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:26.000241 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:47:28.210451 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:28.210346 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:28.210451 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:28.210414 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:28.222720 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:28.222697 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:28.963580 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:28.963546 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hn88h" Jul 22 01:47:30.006871 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:30.006842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:32.542420 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:32.542384 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:47:47.600040 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.600011 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72"] Jul 22 01:47:47.603071 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.603055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.606816 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.606790 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 22 01:47:47.620593 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.620568 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72"] Jul 22 01:47:47.645198 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.645158 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:47.684820 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.684790 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.684993 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.684830 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.684993 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.684943 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.684993 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.684987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.685117 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.685008 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.685117 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.685028 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9jt\" (UniqueName: \"kubernetes.io/projected/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-kube-api-access-6t9jt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.719746 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.719711 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:47.785669 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.785636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.785848 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.785676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.785848 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.785698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.785848 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.785785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9jt\" (UniqueName: \"kubernetes.io/projected/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-kube-api-access-6t9jt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.785848 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.785821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.785848 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.785846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.786174 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.786042 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.786174 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.786110 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.786274 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.786259 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.788006 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.787983 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.788204 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.788187 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.795507 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.795488 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9jt\" (UniqueName: \"kubernetes.io/projected/8e443d2c-6456-49b9-ba3f-fe5a56b936dd-kube-api-access-6t9jt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbq72\" (UID: \"8e443d2c-6456-49b9-ba3f-fe5a56b936dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:47.912025 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:47.911936 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:48.040913 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:48.040870 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72"] Jul 22 01:47:48.043390 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:47:48.043362 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e443d2c_6456_49b9_ba3f_fe5a56b936dd.slice/crio-633388114cd29f1c4529792e3f8355cf5bd66937701348c96814d1c21a1aa20c WatchSource:0}: Error finding container 633388114cd29f1c4529792e3f8355cf5bd66937701348c96814d1c21a1aa20c: Status 404 returned error can't find the container with id 633388114cd29f1c4529792e3f8355cf5bd66937701348c96814d1c21a1aa20c Jul 22 01:47:48.044653 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:48.044634 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:47:49.029076 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:49.029040 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" event={"ID":"8e443d2c-6456-49b9-ba3f-fe5a56b936dd","Type":"ContainerStarted","Data":"5c70a96adfc278c4866d55703945843d808617d9e26832068f8dc117c772676d"} Jul 22 01:47:49.029472 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:49.029086 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" event={"ID":"8e443d2c-6456-49b9-ba3f-fe5a56b936dd","Type":"ContainerStarted","Data":"633388114cd29f1c4529792e3f8355cf5bd66937701348c96814d1c21a1aa20c"} Jul 22 01:47:49.049344 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:49.049289 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" podStartSLOduration=1.8890817260000001 podStartE2EDuration="2.049271087s" totalImagesPullingTime="160.189361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:49.047559988 +0000 UTC m=+643.545001345" watchObservedRunningTime="2026-07-22 01:47:49.049271087 +0000 UTC m=+643.546712436" Jul 22 01:47:50.194029 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:50.193996 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:47:57.913080 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:57.913039 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:57.913703 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:57.913103 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:57.925583 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:57.925561 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:47:58.076001 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:47:58.075974 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbq72" Jul 22 01:49:15.684235 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:15.684150 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:15.741288 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:15.741255 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:24.491467 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:24.491431 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:24.592568 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:24.592513 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:28.683902 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.683869 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f54c4b9cf-tggs9"] Jul 22 01:49:28.686885 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.686869 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" Jul 22 01:49:28.689918 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.689893 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 22 01:49:28.689918 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.689907 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7qvb4\"" Jul 22 01:49:28.702423 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.702398 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f54c4b9cf-tggs9"] Jul 22 01:49:28.751868 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.751826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwq65\" (UniqueName: \"kubernetes.io/projected/010629d1-ad00-4592-8ed7-528e9e8be5ef-kube-api-access-mwq65\") pod \"authorino-6f54c4b9cf-tggs9\" (UID: \"010629d1-ad00-4592-8ed7-528e9e8be5ef\") " pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" Jul 22 01:49:28.752064 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.751898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/010629d1-ad00-4592-8ed7-528e9e8be5ef-tls-cert\") pod \"authorino-6f54c4b9cf-tggs9\" (UID: \"010629d1-ad00-4592-8ed7-528e9e8be5ef\") " pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" Jul 22 01:49:28.852410 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.852364 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/010629d1-ad00-4592-8ed7-528e9e8be5ef-tls-cert\") pod \"authorino-6f54c4b9cf-tggs9\" (UID: \"010629d1-ad00-4592-8ed7-528e9e8be5ef\") " pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" Jul 22 01:49:28.852602 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.852437 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwq65\" (UniqueName: \"kubernetes.io/projected/010629d1-ad00-4592-8ed7-528e9e8be5ef-kube-api-access-mwq65\") pod \"authorino-6f54c4b9cf-tggs9\" (UID: \"010629d1-ad00-4592-8ed7-528e9e8be5ef\") " pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" Jul 22 01:49:28.854791 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.854772 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/010629d1-ad00-4592-8ed7-528e9e8be5ef-tls-cert\") pod \"authorino-6f54c4b9cf-tggs9\" (UID: \"010629d1-ad00-4592-8ed7-528e9e8be5ef\") " pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" Jul 22 01:49:28.861426 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.861395 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwq65\" (UniqueName: \"kubernetes.io/projected/010629d1-ad00-4592-8ed7-528e9e8be5ef-kube-api-access-mwq65\") pod \"authorino-6f54c4b9cf-tggs9\" (UID: \"010629d1-ad00-4592-8ed7-528e9e8be5ef\") " pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" Jul 22 01:49:28.995885 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:28.995817 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" Jul 22 01:49:29.027619 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:29.027580 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:29.123310 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:29.123203 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f54c4b9cf-tggs9"] Jul 22 01:49:29.125716 ip-10-0-130-174 kubenswrapper[2570]: W0722 01:49:29.125686 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod010629d1_ad00_4592_8ed7_528e9e8be5ef.slice/crio-096bdd9bcd75c02b76f3a0ea5deaaebb3bcd7e7c5f9a8c402e86e25755a065e4 WatchSource:0}: Error finding container 096bdd9bcd75c02b76f3a0ea5deaaebb3bcd7e7c5f9a8c402e86e25755a065e4: Status 404 returned error can't find the container with id 096bdd9bcd75c02b76f3a0ea5deaaebb3bcd7e7c5f9a8c402e86e25755a065e4 Jul 22 01:49:29.126909 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:29.126882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:29.421143 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:29.421106 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" event={"ID":"010629d1-ad00-4592-8ed7-528e9e8be5ef","Type":"ContainerStarted","Data":"096bdd9bcd75c02b76f3a0ea5deaaebb3bcd7e7c5f9a8c402e86e25755a065e4"} Jul 22 01:49:30.226383 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:30.226339 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:30.326626 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:30.326593 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:30.427067 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:30.427031 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" event={"ID":"010629d1-ad00-4592-8ed7-528e9e8be5ef","Type":"ContainerStarted","Data":"210d8f387a2e6824e3d709fbf531add31e74ec4d26e01b52ad33eff4aec9a7e7"} Jul 22 01:49:30.453065 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:30.453008 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f54c4b9cf-tggs9" podStartSLOduration=1.979267683 podStartE2EDuration="2.452989887s" totalImagesPullingTime="473.722204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:30.445274989 +0000 UTC m=+744.942716341" watchObservedRunningTime="2026-07-22 01:49:30.452989887 +0000 UTC m=+744.950431243" Jul 22 01:49:57.412481 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:57.412448 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:49:59.960885 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:49:59.960854 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:50:08.892294 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:50:08.892259 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:50:18.598189 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:50:18.598154 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:51:35.128920 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:51:35.128885 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:37.320841 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:51:37.320812 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:42.361602 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:51:42.361568 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:59.017534 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:51:59.017449 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:52:03.033542 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:03.033505 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:03.133615 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:03.133580 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:05.841156 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:05.841127 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:52:05.841653 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:05.841603 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:52:11.592558 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:11.592527 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:52:16.389988 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:16.389958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:16.484891 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:16.484860 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:16.832725 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:16.832650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:22.530973 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:22.530935 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:22.636321 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:22.636283 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:22.731400 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:22.731365 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:22.911729 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:22.911700 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:29.140596 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:29.140563 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:29.241767 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:29.241740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:44.993279 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:44.993243 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:52:54.188632 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:52:54.188598 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:53:03.296817 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:53:03.296787 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:53:06.574268 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:53:06.574233 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:07.997109 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:53:07.997077 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:08.592569 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:53:08.592530 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:53:18.793824 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:53:18.793792 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:53:29.194959 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:53:29.194877 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:53:41.000999 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:53:41.000964 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:53:50.986324 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:53:50.986288 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:54:13.591286 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:54:13.591247 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:54:22.493489 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:54:22.493450 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:54:46.399659 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:54:46.399611 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:54:58.788036 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:54:58.787960 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:55:14.095512 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:55:14.095475 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:55:23.093928 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:55:23.093886 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:55:32.701360 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:55:32.701323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:55:41.992576 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:55:41.992544 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:55:50.889472 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:55:50.889440 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:56:45.588375 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:56:45.588340 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:57:05.863949 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:57:05.863922 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:57:05.865741 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:57:05.865717 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 01:57:28.830865 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:57:28.830819 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:57:43.193468 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:57:43.193437 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:57:52.183780 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:57:52.183741 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:58:04.587862 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:58:04.587772 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:58:15.992074 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:58:15.992034 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:58:27.598247 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:58:27.598202 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:58:38.089298 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:58:38.089257 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:58:45.395516 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:58:45.395483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:46.610010 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:58:46.609973 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:47.405680 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:58:47.405648 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:58:58.093593 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:58:58.093559 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:59:08.490784 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:59:08.490743 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:59:19.106707 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:59:19.106667 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:59:30.623435 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:59:30.623359 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 01:59:45.311158 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:59:45.311125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 01:59:53.048675 ip-10-0-130-174 kubenswrapper[2570]: I0722 01:59:53.048639 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:00:02.823697 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:02.823661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:00:12.619661 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.619630 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp"] Jul 22 02:00:12.622629 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.622612 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.625620 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.625593 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-374f5ecf-kserve-self-signed-certs\"" Jul 22 02:00:12.683546 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.683515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-model-cache\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.683546 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.683551 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngfn\" (UniqueName: \"kubernetes.io/projected/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-kube-api-access-vngfn\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.683744 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.683576 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tls-certs\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.683744 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.683674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-home\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.683744 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.683701 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-dshm\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.683744 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.683729 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tmp-dir\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.696726 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.696694 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp"] Jul 22 02:00:12.700707 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.700684 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:00:12.784164 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784126 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-model-cache\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.784164 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngfn\" (UniqueName: \"kubernetes.io/projected/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-kube-api-access-vngfn\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.784409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tls-certs\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.784409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-home\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.784409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784296 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-dshm\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.784409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tmp-dir\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.784650 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784629 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-model-cache\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.784708 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784668 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-home\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.784778 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.784736 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tmp-dir\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.786662 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.786638 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-dshm\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.786798 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.786780 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tls-certs\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.801525 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.801493 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngfn\" (UniqueName: \"kubernetes.io/projected/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-kube-api-access-vngfn\") pod \"e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:12.931508 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:12.931421 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:13.071271 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:00:13.071240 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec9b0c46_dc0a_46d6_a747_a7adc4938ad7.slice/crio-9914b2744172f825edcfd7e97f48ad5e5d1a47ac7ad189e689ce4beac29fbcd3 WatchSource:0}: Error finding container 9914b2744172f825edcfd7e97f48ad5e5d1a47ac7ad189e689ce4beac29fbcd3: Status 404 returned error can't find the container with id 9914b2744172f825edcfd7e97f48ad5e5d1a47ac7ad189e689ce4beac29fbcd3 Jul 22 02:00:13.073110 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:13.073087 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp"] Jul 22 02:00:14.005680 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:14.005643 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" event={"ID":"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7","Type":"ContainerStarted","Data":"9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab"} Jul 22 02:00:14.005680 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:14.005685 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" event={"ID":"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7","Type":"ContainerStarted","Data":"9914b2744172f825edcfd7e97f48ad5e5d1a47ac7ad189e689ce4beac29fbcd3"} Jul 22 02:00:14.029841 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:14.029794 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" podStartSLOduration=2.029777727 podStartE2EDuration="2.029777727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:00:14.028925559 +0000 UTC m=+1388.526366928" watchObservedRunningTime="2026-07-22 02:00:14.029777727 +0000 UTC m=+1388.527219084" Jul 22 02:00:15.675889 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:15.675841 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:00:22.931701 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:22.931644 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:22.931701 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:22.931714 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:22.944138 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:22.944105 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:23.058252 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.058206 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:00:23.101749 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.101716 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj"] Jul 22 02:00:23.104473 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.104451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.107288 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.107264 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-374f5ecf-kserve-self-signed-certs\"" Jul 22 02:00:23.119700 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.119672 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj"] Jul 22 02:00:23.163309 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.163275 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88bddab-7583-4082-8bae-87965195aa2a-tls-certs\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.163456 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.163319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-model-cache\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.163456 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.163404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvvt\" (UniqueName: \"kubernetes.io/projected/e88bddab-7583-4082-8bae-87965195aa2a-kube-api-access-6gvvt\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.163456 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.163432 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-dshm\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.163559 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.163464 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-tmp-dir\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.163593 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.163549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-home\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.264440 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88bddab-7583-4082-8bae-87965195aa2a-tls-certs\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.264440 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264421 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-model-cache\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.264661 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264470 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvvt\" (UniqueName: \"kubernetes.io/projected/e88bddab-7583-4082-8bae-87965195aa2a-kube-api-access-6gvvt\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.264661 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264497 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-dshm\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.264661 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-tmp-dir\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.264661 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-home\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.265162 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264912 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-model-cache\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.265162 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264956 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-home\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.265162 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.264995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-tmp-dir\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.267075 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.267053 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-dshm\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.267174 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.267112 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88bddab-7583-4082-8bae-87965195aa2a-tls-certs\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.276627 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.276598 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvvt\" (UniqueName: \"kubernetes.io/projected/e88bddab-7583-4082-8bae-87965195aa2a-kube-api-access-6gvvt\") pod \"e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.347367 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.347335 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:00:23.415268 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.415234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:23.555420 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:23.555349 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj"] Jul 22 02:00:23.558400 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:00:23.558367 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode88bddab_7583_4082_8bae_87965195aa2a.slice/crio-a12ce2e26ca91668d36091eac8d5196768a974994727cba5e2df936177e3aefc WatchSource:0}: Error finding container a12ce2e26ca91668d36091eac8d5196768a974994727cba5e2df936177e3aefc: Status 404 returned error can't find the container with id a12ce2e26ca91668d36091eac8d5196768a974994727cba5e2df936177e3aefc Jul 22 02:00:24.052591 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:24.052552 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" event={"ID":"e88bddab-7583-4082-8bae-87965195aa2a","Type":"ContainerStarted","Data":"9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727"} Jul 22 02:00:24.052979 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:24.052599 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" event={"ID":"e88bddab-7583-4082-8bae-87965195aa2a","Type":"ContainerStarted","Data":"a12ce2e26ca91668d36091eac8d5196768a974994727cba5e2df936177e3aefc"} Jul 22 02:00:24.082683 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:24.082637 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" podStartSLOduration=1.082620985 podStartE2EDuration="1.082620985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:00:24.0815921 +0000 UTC m=+1398.579033460" watchObservedRunningTime="2026-07-22 02:00:24.082620985 +0000 UTC m=+1398.580062341" Jul 22 02:00:33.415979 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:33.415936 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:33.416403 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:33.416236 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:33.428165 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:33.428143 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:34.103498 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:34.103466 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:00:35.189262 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:35.189208 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:00:50.641566 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:50.641534 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp"] Jul 22 02:00:50.668199 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:50.668172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:00:51.852326 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:51.852295 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj"] Jul 22 02:00:51.874511 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:51.874486 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:00:54.297618 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:00:54.297582 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:01:05.661071 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:05.660986 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" podUID="ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" containerName="main" containerID="cri-o://9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab" gracePeriod=45 Jul 22 02:01:05.907396 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:05.907369 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:01:06.029941 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.029861 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-model-cache\") pod \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " Jul 22 02:01:06.029941 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.029902 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-dshm\") pod \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " Jul 22 02:01:06.029941 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.029923 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-home\") pod \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " Jul 22 02:01:06.030240 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.029945 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tls-certs\") pod \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " Jul 22 02:01:06.030240 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.030026 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vngfn\" (UniqueName: \"kubernetes.io/projected/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-kube-api-access-vngfn\") pod \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " Jul 22 02:01:06.030240 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.030080 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tmp-dir\") pod \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\" (UID: \"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7\") " Jul 22 02:01:06.030240 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.030178 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-home" (OuterVolumeSpecName: "home") pod "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" (UID: "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:01:06.030240 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.030181 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" (UID: "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:01:06.030502 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.030365 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:06.030502 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.030377 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" (UID: "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:01:06.030502 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.030385 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:06.032228 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.032186 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-kube-api-access-vngfn" (OuterVolumeSpecName: "kube-api-access-vngfn") pod "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" (UID: "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7"). InnerVolumeSpecName "kube-api-access-vngfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:01:06.032343 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.032250 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-dshm" (OuterVolumeSpecName: "dshm") pod "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" (UID: "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:01:06.032343 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.032260 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" (UID: "ec9b0c46-dc0a-46d6-a747-a7adc4938ad7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:01:06.131640 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.131608 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:06.131640 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.131635 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:06.131640 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.131645 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:06.131859 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.131653 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vngfn\" (UniqueName: \"kubernetes.io/projected/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7-kube-api-access-vngfn\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:06.232288 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.232251 2570 generic.go:356] "Generic (PLEG): container finished" podID="ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" containerID="9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab" exitCode=0 Jul 22 02:01:06.232423 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.232301 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" Jul 22 02:01:06.232423 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.232320 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" event={"ID":"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7","Type":"ContainerDied","Data":"9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab"} Jul 22 02:01:06.232423 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.232358 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp" event={"ID":"ec9b0c46-dc0a-46d6-a747-a7adc4938ad7","Type":"ContainerDied","Data":"9914b2744172f825edcfd7e97f48ad5e5d1a47ac7ad189e689ce4beac29fbcd3"} Jul 22 02:01:06.232423 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.232381 2570 scope.go:117] "RemoveContainer" containerID="9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab" Jul 22 02:01:06.240154 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.240134 2570 scope.go:117] "RemoveContainer" containerID="9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab" Jul 22 02:01:06.240432 ip-10-0-130-174 kubenswrapper[2570]: E0722 02:01:06.240412 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab\": container with ID starting with 9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab not found: ID does not exist" containerID="9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab" Jul 22 02:01:06.240506 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.240447 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab"} err="failed to get container status \"9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab\": rpc error: code = NotFound desc = could not find container \"9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab\": container with ID starting with 9a909431f7c8a1afc32b018145b254b00a916ffa3ec619f56823137d309a74ab not found: ID does not exist" Jul 22 02:01:06.253437 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.253409 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp"] Jul 22 02:01:06.257818 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.257799 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-374f5ecf-kserve-69b74fccc9-gkgmp"] Jul 22 02:01:06.872022 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:06.871980 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" podUID="e88bddab-7583-4082-8bae-87965195aa2a" containerName="main" containerID="cri-o://9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727" gracePeriod=45 Jul 22 02:01:07.114800 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.114773 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:01:07.137722 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.137658 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gvvt\" (UniqueName: \"kubernetes.io/projected/e88bddab-7583-4082-8bae-87965195aa2a-kube-api-access-6gvvt\") pod \"e88bddab-7583-4082-8bae-87965195aa2a\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " Jul 22 02:01:07.137722 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.137692 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-model-cache\") pod \"e88bddab-7583-4082-8bae-87965195aa2a\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " Jul 22 02:01:07.137901 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.137747 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88bddab-7583-4082-8bae-87965195aa2a-tls-certs\") pod \"e88bddab-7583-4082-8bae-87965195aa2a\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " Jul 22 02:01:07.137901 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.137782 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-dshm\") pod \"e88bddab-7583-4082-8bae-87965195aa2a\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " Jul 22 02:01:07.137901 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.137811 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-home\") pod \"e88bddab-7583-4082-8bae-87965195aa2a\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " Jul 22 02:01:07.137901 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.137835 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-tmp-dir\") pod \"e88bddab-7583-4082-8bae-87965195aa2a\" (UID: \"e88bddab-7583-4082-8bae-87965195aa2a\") " Jul 22 02:01:07.138111 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.138087 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e88bddab-7583-4082-8bae-87965195aa2a" (UID: "e88bddab-7583-4082-8bae-87965195aa2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:01:07.138167 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.138101 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-home" (OuterVolumeSpecName: "home") pod "e88bddab-7583-4082-8bae-87965195aa2a" (UID: "e88bddab-7583-4082-8bae-87965195aa2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:01:07.138253 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.138201 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e88bddab-7583-4082-8bae-87965195aa2a" (UID: "e88bddab-7583-4082-8bae-87965195aa2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:01:07.139918 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.139894 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-dshm" (OuterVolumeSpecName: "dshm") pod "e88bddab-7583-4082-8bae-87965195aa2a" (UID: "e88bddab-7583-4082-8bae-87965195aa2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:01:07.140013 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.139988 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88bddab-7583-4082-8bae-87965195aa2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e88bddab-7583-4082-8bae-87965195aa2a" (UID: "e88bddab-7583-4082-8bae-87965195aa2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:01:07.140098 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.140079 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88bddab-7583-4082-8bae-87965195aa2a-kube-api-access-6gvvt" (OuterVolumeSpecName: "kube-api-access-6gvvt") pod "e88bddab-7583-4082-8bae-87965195aa2a" (UID: "e88bddab-7583-4082-8bae-87965195aa2a"). InnerVolumeSpecName "kube-api-access-6gvvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:01:07.238461 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238434 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88bddab-7583-4082-8bae-87965195aa2a-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:07.238461 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238459 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:07.238694 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238470 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:07.238694 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238483 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:07.238694 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238495 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gvvt\" (UniqueName: \"kubernetes.io/projected/e88bddab-7583-4082-8bae-87965195aa2a-kube-api-access-6gvvt\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:07.238694 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238509 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88bddab-7583-4082-8bae-87965195aa2a-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:01:07.238694 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238571 2570 generic.go:356] "Generic (PLEG): container finished" podID="e88bddab-7583-4082-8bae-87965195aa2a" containerID="9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727" exitCode=0 Jul 22 02:01:07.238694 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238648 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" event={"ID":"e88bddab-7583-4082-8bae-87965195aa2a","Type":"ContainerDied","Data":"9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727"} Jul 22 02:01:07.238694 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" event={"ID":"e88bddab-7583-4082-8bae-87965195aa2a","Type":"ContainerDied","Data":"a12ce2e26ca91668d36091eac8d5196768a974994727cba5e2df936177e3aefc"} Jul 22 02:01:07.239237 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238705 2570 scope.go:117] "RemoveContainer" containerID="9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727" Jul 22 02:01:07.239237 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.238713 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj" Jul 22 02:01:07.251365 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.251347 2570 scope.go:117] "RemoveContainer" containerID="9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727" Jul 22 02:01:07.251632 ip-10-0-130-174 kubenswrapper[2570]: E0722 02:01:07.251611 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727\": container with ID starting with 9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727 not found: ID does not exist" containerID="9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727" Jul 22 02:01:07.251682 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.251639 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727"} err="failed to get container status \"9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727\": rpc error: code = NotFound desc = could not find container \"9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727\": container with ID starting with 9e34e1767eb6b299391201236bc3bebea80a7739bed46f9fa56da82932c36727 not found: ID does not exist" Jul 22 02:01:07.262963 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.262940 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj"] Jul 22 02:01:07.266979 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.266959 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-374f5ecf-kserve-6f8f69975-c8vgj"] Jul 22 02:01:07.597651 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.597577 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e88bddab-7583-4082-8bae-87965195aa2a" path="/var/lib/kubelet/pods/e88bddab-7583-4082-8bae-87965195aa2a/volumes" Jul 22 02:01:07.597944 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.597931 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" path="/var/lib/kubelet/pods/ec9b0c46-dc0a-46d6-a747-a7adc4938ad7/volumes" Jul 22 02:01:07.891595 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:07.891566 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:01:16.697140 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:16.697101 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:01:27.792991 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:27.792948 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:01:36.693882 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:36.693837 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:01:45.894465 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:45.894424 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:01:56.197151 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:01:56.197113 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:02:05.294139 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:02:05.294097 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:02:05.890681 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:02:05.890651 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:02:05.892115 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:02:05.892096 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:02:14.486654 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:02:14.486621 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:02:27.691125 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:02:27.691087 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:02:36.790488 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:02:36.790458 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:02:46.092315 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:02:46.092280 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:02:58.282880 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:02:58.282846 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:03:09.387008 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:03:09.386971 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:03:17.096011 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:03:17.095974 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:03:26.592060 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:03:26.592019 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:03:42.890970 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:03:42.890931 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:03:52.096682 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:03:52.096647 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:03:55.439366 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:03:55.439326 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:05.987609 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:05.987532 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:04:14.016694 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:14.016650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:17.143303 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:17.143260 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:17.247020 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:17.246988 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:17.648489 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:17.648455 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:18.196721 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:18.196688 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:04:23.816671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:23.816633 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:23.904793 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:23.904754 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:24.063201 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:24.063162 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:24.380281 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:24.380249 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:25.404635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:25.404604 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:26.732696 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:26.732660 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:26.832740 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:26.832698 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:26.934675 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:26.934635 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:27.032724 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:27.032636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:27.424154 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:27.424121 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:27.524640 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:27.524596 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:34.987742 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:34.987700 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:04:51.825938 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:51.825901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:51.880509 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:04:51.880470 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:02.982762 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:02.982723 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:05.496047 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:05.496005 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:05:07.498439 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:07.498405 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:07.618945 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:07.618910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:07.693314 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:07.693275 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:07.930742 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:07.930704 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:08.219844 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:08.219758 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:08.318755 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:08.318718 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:08.441587 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:08.441554 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:08.781707 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:08.781672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:09.130605 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:09.130572 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:11.072492 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:11.072460 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:11.191719 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:11.191688 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:11.690890 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:11.690858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:11.791513 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:11.791479 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:11.945900 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:11.945810 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:18.390316 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:18.390286 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:05:46.698569 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:46.698534 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:49.490614 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:49.490583 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:05:50.720958 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:50.720916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:51.237902 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:51.237863 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:51.407929 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:51.407895 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:51.654776 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:51.654737 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:51.918035 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:51.917935 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:52.229717 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:52.229631 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:52.980745 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:52.980712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:54.263515 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:54.263478 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:54.346905 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:54.346875 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:54.415733 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:54.415685 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:54.518645 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:54.518563 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:54.896644 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:54.896617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:05:54.998181 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:05:54.998153 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:01.795451 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:01.795417 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:06:30.654272 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:30.654235 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:33.494035 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:33.493996 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:06:35.275389 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:35.275348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:36.528014 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:36.527980 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:36.766593 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:36.766560 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:37.046798 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:37.046765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:37.328295 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:37.328204 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:37.976412 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:37.976376 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:46.176960 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:46.176920 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:46.280954 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:46.280919 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:46.375639 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:46.375600 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:46.476618 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:46.476541 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:46.884615 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:46.884579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:46.984235 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:46.984194 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:50.888596 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:50.888567 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:06:58.129686 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:58.129599 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:58.231789 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:06:58.231756 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:05.917231 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:05.917184 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:07:05.918344 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:05.918322 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:07:07.960839 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:07.960796 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:08.061807 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:08.061775 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:21.677628 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:21.677589 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:24.709057 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:24.709020 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:07:26.240551 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:26.240522 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:26.421267 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:26.421235 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:26.666690 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:26.666621 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:27.178633 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:27.178595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:27.273891 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:27.273857 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:27.380759 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:27.380729 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:28.904755 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:28.904715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:34.960896 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:34.960858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:35.065349 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:35.065287 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:35.161789 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:35.161736 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:35.261747 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:35.261657 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:35.722153 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:07:35.722113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:01.824254 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:01.824206 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:08.540467 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:08.540428 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:08.613299 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:08.613270 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:08.704372 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:08.704338 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:24.701415 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:24.701380 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:30.753938 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:30.753847 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:30.859912 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:30.859877 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:30.954533 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:30.954500 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:41.533501 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:41.533443 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:42.342951 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:42.342918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:49.124310 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:49.124275 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:49.254075 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:49.254035 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:49.353589 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:49.353557 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:49.454661 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:49.454569 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:50.234201 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:50.234165 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:50.337766 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:50.337731 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:50.460295 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:50.460262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:52.148373 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:52.148344 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:52.788160 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:52.788121 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:08:58.078901 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:58.078871 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:08:59.490794 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:08:59.490758 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:04.697948 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:04.697908 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:09:05.344009 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:05.343962 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:05.446606 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:05.446570 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:05.609723 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:05.609689 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:05.962039 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:05.961902 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:06.352142 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:06.352106 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:08.905691 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:08.905650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:08.997832 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:08.997785 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:09.095466 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:09.095434 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:09.482260 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:09.482187 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:09.586362 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:09.586329 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:09.682757 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:09.682722 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:13.387597 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:13.387559 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:14.079120 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:14.079081 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:14.179378 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:14.179345 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:14.515208 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:14.515166 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:14.889076 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:14.889041 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:09:20.825671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:20.825634 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:20.925675 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:20.925647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:31.771989 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:31.771959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:34.597578 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:34.597547 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:09:36.289563 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:36.289526 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:36.714757 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:36.714718 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:42.591047 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:42.591006 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:42.689990 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:42.689954 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:42.873971 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:42.873934 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:43.119798 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:43.119764 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:43.237384 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:43.237296 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:43.337687 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:43.337656 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:43.485796 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:43.485765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:49.515078 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:49.515047 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:49.579061 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:49.579033 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:49.686611 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:49.686576 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:49.791266 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:49.791167 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:49.888797 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:49.888763 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:55.838639 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:55.838608 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:55.908422 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:09:55.908390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:02.761071 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:02.760985 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:03.096906 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:03.096824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:03.191921 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:03.191881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:06.249231 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:06.249172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:06.355583 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:06.355553 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:06.800290 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:06.800260 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:06.904170 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:06.904127 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:10.628825 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:10.628785 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:12.392005 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:12.391969 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:10:13.499107 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.499070 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng"] Jul 22 02:10:13.499535 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.499397 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88bddab-7583-4082-8bae-87965195aa2a" containerName="main" Jul 22 02:10:13.499535 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.499410 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88bddab-7583-4082-8bae-87965195aa2a" containerName="main" Jul 22 02:10:13.499535 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.499421 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" containerName="main" Jul 22 02:10:13.499535 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.499428 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" containerName="main" Jul 22 02:10:13.499535 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.499476 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9b0c46-dc0a-46d6-a747-a7adc4938ad7" containerName="main" Jul 22 02:10:13.499535 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.499486 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88bddab-7583-4082-8bae-87965195aa2a" containerName="main" Jul 22 02:10:13.502422 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.502399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.504954 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.504929 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb9ad51b\"/\"openshift-service-ca.crt\"" Jul 22 02:10:13.505072 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.504930 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb9ad51b\"/\"default-dockercfg-bzvdx\"" Jul 22 02:10:13.505849 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.505826 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb9ad51b\"/\"kube-root-ca.crt\"" Jul 22 02:10:13.505961 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.505885 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb9ad51b\"/\"e2e-lifecycle-model-fb9ad51b-kserve-self-signed-certs\"" Jul 22 02:10:13.511408 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.511383 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng"] Jul 22 02:10:13.524797 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.524767 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:13.556353 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.556307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjzw\" (UniqueName: \"kubernetes.io/projected/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-kube-api-access-vmjzw\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.556548 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.556361 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tls-certs\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.556548 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.556403 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-home\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.556548 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.556438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-model-cache\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.556548 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.556464 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-dshm\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.556548 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.556539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tmp-dir\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.656968 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.656928 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjzw\" (UniqueName: \"kubernetes.io/projected/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-kube-api-access-vmjzw\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.657180 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.656979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tls-certs\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.657180 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.657001 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-home\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.657180 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.657029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-model-cache\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.657180 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.657168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-dshm\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.657433 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.657331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tmp-dir\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.657488 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.657455 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-home\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.657542 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.657494 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-model-cache\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.657676 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.657651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tmp-dir\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.659450 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.659428 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-dshm\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.659546 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.659529 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tls-certs\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.665610 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.665590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjzw\" (UniqueName: \"kubernetes.io/projected/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-kube-api-access-vmjzw\") pod \"e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.811844 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.811743 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:13.950106 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:13.950068 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng"] Jul 22 02:10:13.953257 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:10:13.953226 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d006ef4_4d2c_4f1f_8343_fbf44cedb165.slice/crio-8efcc4c4bc5917df2f3559dafff021a17397adce3f67c84ad624e1e18c9d16d6 WatchSource:0}: Error finding container 8efcc4c4bc5917df2f3559dafff021a17397adce3f67c84ad624e1e18c9d16d6: Status 404 returned error can't find the container with id 8efcc4c4bc5917df2f3559dafff021a17397adce3f67c84ad624e1e18c9d16d6 Jul 22 02:10:14.447102 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:14.447056 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" event={"ID":"4d006ef4-4d2c-4f1f-8343-fbf44cedb165","Type":"ContainerStarted","Data":"1699cbedd58126c83e0e2477d5b8a0d44ed81e5b18f88c715317815136f1c195"} Jul 22 02:10:14.447290 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:14.447102 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" event={"ID":"4d006ef4-4d2c-4f1f-8343-fbf44cedb165","Type":"ContainerStarted","Data":"8efcc4c4bc5917df2f3559dafff021a17397adce3f67c84ad624e1e18c9d16d6"} Jul 22 02:10:14.465287 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:14.465188 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" podStartSLOduration=1.46517311 podStartE2EDuration="1.46517311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:10:14.46382226 +0000 UTC m=+1988.961263637" watchObservedRunningTime="2026-07-22 02:10:14.46517311 +0000 UTC m=+1988.962614467" Jul 22 02:10:21.989340 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:21.989305 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:10:23.812687 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:23.812657 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:23.813136 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:23.812714 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:23.824979 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:23.824955 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:24.500868 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:24.500834 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:10:30.795025 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:30.794987 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:10:48.286243 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:48.286192 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:10:49.469029 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:49.468998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:54.040283 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:54.040245 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:56.485872 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:56.485835 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:58.692299 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:10:58.692258 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:11:01.472690 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:01.472653 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng"] Jul 22 02:11:01.530267 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:01.530234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:16.492544 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.492503 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" podUID="4d006ef4-4d2c-4f1f-8343-fbf44cedb165" containerName="main" containerID="cri-o://1699cbedd58126c83e0e2477d5b8a0d44ed81e5b18f88c715317815136f1c195" gracePeriod=45 Jul 22 02:11:16.699477 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.699444 2570 generic.go:356] "Generic (PLEG): container finished" podID="4d006ef4-4d2c-4f1f-8343-fbf44cedb165" containerID="1699cbedd58126c83e0e2477d5b8a0d44ed81e5b18f88c715317815136f1c195" exitCode=0 Jul 22 02:11:16.699644 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.699511 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" event={"ID":"4d006ef4-4d2c-4f1f-8343-fbf44cedb165","Type":"ContainerDied","Data":"1699cbedd58126c83e0e2477d5b8a0d44ed81e5b18f88c715317815136f1c195"} Jul 22 02:11:16.734341 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.734316 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:11:16.764493 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764416 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-model-cache\") pod \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " Jul 22 02:11:16.764493 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764449 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-dshm\") pod \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " Jul 22 02:11:16.764686 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764518 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tmp-dir\") pod \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " Jul 22 02:11:16.764686 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764550 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmjzw\" (UniqueName: \"kubernetes.io/projected/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-kube-api-access-vmjzw\") pod \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " Jul 22 02:11:16.764686 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764589 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tls-certs\") pod \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " Jul 22 02:11:16.764686 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764669 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-home\") pod \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\" (UID: \"4d006ef4-4d2c-4f1f-8343-fbf44cedb165\") " Jul 22 02:11:16.764849 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764715 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d006ef4-4d2c-4f1f-8343-fbf44cedb165" (UID: "4d006ef4-4d2c-4f1f-8343-fbf44cedb165"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:16.764849 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764793 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d006ef4-4d2c-4f1f-8343-fbf44cedb165" (UID: "4d006ef4-4d2c-4f1f-8343-fbf44cedb165"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:16.764921 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764893 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:11:16.764921 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.764912 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:11:16.765027 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.765006 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-home" (OuterVolumeSpecName: "home") pod "4d006ef4-4d2c-4f1f-8343-fbf44cedb165" (UID: "4d006ef4-4d2c-4f1f-8343-fbf44cedb165"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:16.767076 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.767035 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d006ef4-4d2c-4f1f-8343-fbf44cedb165" (UID: "4d006ef4-4d2c-4f1f-8343-fbf44cedb165"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:11:16.767200 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.767113 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-dshm" (OuterVolumeSpecName: "dshm") pod "4d006ef4-4d2c-4f1f-8343-fbf44cedb165" (UID: "4d006ef4-4d2c-4f1f-8343-fbf44cedb165"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:16.767200 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.767144 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-kube-api-access-vmjzw" (OuterVolumeSpecName: "kube-api-access-vmjzw") pod "4d006ef4-4d2c-4f1f-8343-fbf44cedb165" (UID: "4d006ef4-4d2c-4f1f-8343-fbf44cedb165"). InnerVolumeSpecName "kube-api-access-vmjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:11:16.865635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.865580 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:11:16.865635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.865626 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:11:16.865635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.865636 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjzw\" (UniqueName: \"kubernetes.io/projected/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-kube-api-access-vmjzw\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:11:16.865635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:16.865646 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d006ef4-4d2c-4f1f-8343-fbf44cedb165-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:11:17.705189 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:17.705160 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" Jul 22 02:11:17.705189 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:17.705182 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng" event={"ID":"4d006ef4-4d2c-4f1f-8343-fbf44cedb165","Type":"ContainerDied","Data":"8efcc4c4bc5917df2f3559dafff021a17397adce3f67c84ad624e1e18c9d16d6"} Jul 22 02:11:17.705722 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:17.705251 2570 scope.go:117] "RemoveContainer" containerID="1699cbedd58126c83e0e2477d5b8a0d44ed81e5b18f88c715317815136f1c195" Jul 22 02:11:17.735625 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:17.735596 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng"] Jul 22 02:11:17.738387 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:17.738361 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fb9ad51b/e2e-lifecycle-model-fb9ad51b-kserve-5cfdcbf5d7-fx9ng"] Jul 22 02:11:19.598460 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:19.598425 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d006ef4-4d2c-4f1f-8343-fbf44cedb165" path="/var/lib/kubelet/pods/4d006ef4-4d2c-4f1f-8343-fbf44cedb165/volumes" Jul 22 02:11:38.345840 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:38.345754 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:38.751319 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:38.751284 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:44.680355 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:44.680316 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:44.834630 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:44.834592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:45.002636 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:45.002551 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:45.250756 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:45.250716 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:46.028883 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:46.028849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:46.338529 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:46.338438 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:46.438355 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:46.438304 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:46.539681 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:46.539637 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:47.372007 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:47.371974 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:47.468883 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:47.468853 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:47.565564 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:47.565533 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:48.726732 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:48.726699 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:11:49.077074 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:49.076983 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:55.388751 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:55.388710 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:55.456409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:55.456373 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:56.103333 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:56.103288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:58.604387 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:58.604349 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:11:59.072980 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:59.072893 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:59.173113 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:11:59.173072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:02.399142 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:02.399104 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:02.500881 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:02.500828 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:02.609299 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:02.609262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:05.938402 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:05.938377 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:12:05.940011 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:05.939991 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:12:13.162758 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:13.162727 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:13.530254 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:13.530147 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:17.943919 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:17.943887 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:18.043514 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:18.043482 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:25.497109 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:25.493089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:25.565783 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:25.565740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:32.866932 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:32.866896 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:34.552634 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:34.552593 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:41.377426 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:41.377387 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:41.478605 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:41.478563 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:41.765726 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:41.765636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:42.124950 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:42.124897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:42.283589 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:42.283555 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:44.921033 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:44.920996 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:45.021023 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:45.020980 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:45.119273 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:45.119234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:45.242139 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:45.242053 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:45.631404 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:45.631372 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:45.726686 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:45.726653 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:49.474854 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:49.474821 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:49.822965 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:49.822877 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:50.202680 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:50.202633 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:50.298947 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:50.298912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:50.598589 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:50.598504 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:12:53.024636 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:53.024606 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:53.124514 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:53.124483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:53.224678 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:53.224640 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:53.440431 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:53.440398 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:53.806607 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:53.806479 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:53.906785 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:53.906745 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:57.605782 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:12:57.605742 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:04.600524 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:04.600492 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:13:07.646426 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:07.646395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:07.747918 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:07.747880 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:15.759923 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:15.759888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:15.861528 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:15.861487 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:17.105180 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.105146 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84"] Jul 22 02:13:17.105587 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.105460 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d006ef4-4d2c-4f1f-8343-fbf44cedb165" containerName="main" Jul 22 02:13:17.105587 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.105473 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d006ef4-4d2c-4f1f-8343-fbf44cedb165" containerName="main" Jul 22 02:13:17.105587 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.105532 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d006ef4-4d2c-4f1f-8343-fbf44cedb165" containerName="main" Jul 22 02:13:17.108252 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.108234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.110635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.110608 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b0ac34\"/\"kube-root-ca.crt\"" Jul 22 02:13:17.110779 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.110761 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b0ac34\"/\"openshift-service-ca.crt\"" Jul 22 02:13:17.111807 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.111789 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b0ac34\"/\"default-dockercfg-vs45j\"" Jul 22 02:13:17.111904 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.111883 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b0ac34\"/\"test-model-b0ac34-kserve-self-signed-certs\"" Jul 22 02:13:17.119397 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.119377 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84"] Jul 22 02:13:17.143909 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.143875 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tmp-dir\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.143909 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.143912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-home\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.144123 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.143967 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-dshm\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.144123 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.143989 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tls-certs\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.144123 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.144038 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-model-cache\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.144123 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.144061 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xcmq\" (UniqueName: \"kubernetes.io/projected/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-kube-api-access-8xcmq\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.144446 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.144426 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:17.244578 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.244538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-model-cache\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.244578 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.244581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xcmq\" (UniqueName: \"kubernetes.io/projected/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-kube-api-access-8xcmq\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.244793 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.244617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tmp-dir\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.244793 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.244643 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-home\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.244793 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.244689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-dshm\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.244793 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.244723 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tls-certs\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.245042 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.245016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-model-cache\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.245079 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.245028 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-home\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.245113 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.245078 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tmp-dir\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.247037 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.247012 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-dshm\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.247197 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.247180 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tls-certs\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.255500 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.255478 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xcmq\" (UniqueName: \"kubernetes.io/projected/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-kube-api-access-8xcmq\") pod \"test-model-b0ac34-kserve-69c6544f55-f5d84\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.416841 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.416757 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:17.562126 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:17.562092 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84"] Jul 22 02:13:17.565162 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:13:17.565135 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf95966e0_4883_4bcf_a8d2_9e92782fc0b0.slice/crio-2b44ecbaa6bd9223ca947a44f88aee575ee1aea93d6773040839a1f8cdd6d930 WatchSource:0}: Error finding container 2b44ecbaa6bd9223ca947a44f88aee575ee1aea93d6773040839a1f8cdd6d930: Status 404 returned error can't find the container with id 2b44ecbaa6bd9223ca947a44f88aee575ee1aea93d6773040839a1f8cdd6d930 Jul 22 02:13:18.200204 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:18.200161 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" event={"ID":"f95966e0-4883-4bcf-a8d2-9e92782fc0b0","Type":"ContainerStarted","Data":"8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff"} Jul 22 02:13:18.200684 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:18.200207 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" event={"ID":"f95966e0-4883-4bcf-a8d2-9e92782fc0b0","Type":"ContainerStarted","Data":"2b44ecbaa6bd9223ca947a44f88aee575ee1aea93d6773040839a1f8cdd6d930"} Jul 22 02:13:18.218299 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:18.218253 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" podStartSLOduration=1.2182361959999999 podStartE2EDuration="1.218236196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:13:18.217523244 +0000 UTC m=+2172.714964602" watchObservedRunningTime="2026-07-22 02:13:18.218236196 +0000 UTC m=+2172.715677563" Jul 22 02:13:21.892957 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:21.892918 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:13:27.417084 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:27.417045 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:27.417586 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:27.417094 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:27.429879 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:27.429849 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:28.251861 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:28.251832 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:13:48.205907 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.205872 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz"] Jul 22 02:13:48.208696 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.208679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.211970 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.211928 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1ad649\"/\"test-model-1ad649-kserve-self-signed-certs\"" Jul 22 02:13:48.211970 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.211930 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1ad649\"/\"openshift-service-ca.crt\"" Jul 22 02:13:48.212264 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.211973 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1ad649\"/\"kube-root-ca.crt\"" Jul 22 02:13:48.212264 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.211990 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1ad649\"/\"default-dockercfg-mvsst\"" Jul 22 02:13:48.220083 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.220058 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz"] Jul 22 02:13:48.295447 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.295412 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:48.304146 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.304107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-dshm\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.304323 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.304163 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ee45f8-6de5-405f-bf30-e496fa78f764-tls-certs\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.304323 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.304191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-model-cache\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.304323 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.304293 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-tmp-dir\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.304479 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.304357 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8l7h\" (UniqueName: \"kubernetes.io/projected/36ee45f8-6de5-405f-bf30-e496fa78f764-kube-api-access-j8l7h\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.304479 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.304402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-home\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.405495 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.405452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ee45f8-6de5-405f-bf30-e496fa78f764-tls-certs\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.405495 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.405496 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-model-cache\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.405727 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.405518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-tmp-dir\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.405727 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.405549 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8l7h\" (UniqueName: \"kubernetes.io/projected/36ee45f8-6de5-405f-bf30-e496fa78f764-kube-api-access-j8l7h\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.405727 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.405571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-home\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.405871 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.405723 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-dshm\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.405932 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.405910 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-model-cache\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.405989 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.405950 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-tmp-dir\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.406042 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.406009 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-home\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.407825 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.407802 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-dshm\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.407973 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.407956 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ee45f8-6de5-405f-bf30-e496fa78f764-tls-certs\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.414525 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.414496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8l7h\" (UniqueName: \"kubernetes.io/projected/36ee45f8-6de5-405f-bf30-e496fa78f764-kube-api-access-j8l7h\") pod \"test-model-1ad649-kserve-775877555f-8zcwz\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.518321 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.518231 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:48.652872 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:48.652840 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz"] Jul 22 02:13:48.656466 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:13:48.656425 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36ee45f8_6de5_405f_bf30_e496fa78f764.slice/crio-50893604e4d58915634500a583b1a38dde7fb0dbd18110e56eaf7b0b8951ab75 WatchSource:0}: Error finding container 50893604e4d58915634500a583b1a38dde7fb0dbd18110e56eaf7b0b8951ab75: Status 404 returned error can't find the container with id 50893604e4d58915634500a583b1a38dde7fb0dbd18110e56eaf7b0b8951ab75 Jul 22 02:13:49.340595 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:49.340559 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" event={"ID":"36ee45f8-6de5-405f-bf30-e496fa78f764","Type":"ContainerStarted","Data":"f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305"} Jul 22 02:13:49.340595 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:49.340599 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" event={"ID":"36ee45f8-6de5-405f-bf30-e496fa78f764","Type":"ContainerStarted","Data":"50893604e4d58915634500a583b1a38dde7fb0dbd18110e56eaf7b0b8951ab75"} Jul 22 02:13:49.359972 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:49.359913 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" podStartSLOduration=1.359895308 podStartE2EDuration="1.359895308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:13:49.359041007 +0000 UTC m=+2203.856482365" watchObservedRunningTime="2026-07-22 02:13:49.359895308 +0000 UTC m=+2203.857336665" Jul 22 02:13:52.003205 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:52.003168 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:13:58.518385 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:58.518348 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:58.518385 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:58.518395 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:58.530778 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:58.530754 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:13:59.390598 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:13:59.390570 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:14:03.172112 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:03.172073 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:07.878622 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:07.878591 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:09.628360 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:09.628327 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:14:28.181979 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:28.181894 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84"] Jul 22 02:14:28.314467 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:28.314431 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:29.199940 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:29.199908 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz"] Jul 22 02:14:29.234683 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:29.234651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:30.235837 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:30.235798 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:30.371951 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:30.371918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:32.905087 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:32.905048 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:14:40.070051 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:40.070015 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:43.201335 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.201283 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" podUID="f95966e0-4883-4bcf-a8d2-9e92782fc0b0" containerName="main" containerID="cri-o://8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff" gracePeriod=45 Jul 22 02:14:43.439382 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.439356 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:14:43.565022 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.564935 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tmp-dir\") pod \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " Jul 22 02:14:43.565022 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.564981 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xcmq\" (UniqueName: \"kubernetes.io/projected/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-kube-api-access-8xcmq\") pod \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " Jul 22 02:14:43.565263 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565031 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-home\") pod \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " Jul 22 02:14:43.565263 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565061 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-dshm\") pod \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " Jul 22 02:14:43.565263 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565098 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-model-cache\") pod \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " Jul 22 02:14:43.565263 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565124 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tls-certs\") pod \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\" (UID: \"f95966e0-4883-4bcf-a8d2-9e92782fc0b0\") " Jul 22 02:14:43.565697 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565246 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f95966e0-4883-4bcf-a8d2-9e92782fc0b0" (UID: "f95966e0-4883-4bcf-a8d2-9e92782fc0b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:14:43.565697 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565327 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-home" (OuterVolumeSpecName: "home") pod "f95966e0-4883-4bcf-a8d2-9e92782fc0b0" (UID: "f95966e0-4883-4bcf-a8d2-9e92782fc0b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:14:43.565697 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565396 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f95966e0-4883-4bcf-a8d2-9e92782fc0b0" (UID: "f95966e0-4883-4bcf-a8d2-9e92782fc0b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:14:43.565697 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565420 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:43.565697 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.565440 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:43.567491 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.567460 2570 generic.go:356] "Generic (PLEG): container finished" podID="f95966e0-4883-4bcf-a8d2-9e92782fc0b0" containerID="8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff" exitCode=0 Jul 22 02:14:43.567491 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.567480 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-dshm" (OuterVolumeSpecName: "dshm") pod "f95966e0-4883-4bcf-a8d2-9e92782fc0b0" (UID: "f95966e0-4883-4bcf-a8d2-9e92782fc0b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:14:43.567671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.567492 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" event={"ID":"f95966e0-4883-4bcf-a8d2-9e92782fc0b0","Type":"ContainerDied","Data":"8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff"} Jul 22 02:14:43.567671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.567517 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" Jul 22 02:14:43.567671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.567543 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84" event={"ID":"f95966e0-4883-4bcf-a8d2-9e92782fc0b0","Type":"ContainerDied","Data":"2b44ecbaa6bd9223ca947a44f88aee575ee1aea93d6773040839a1f8cdd6d930"} Jul 22 02:14:43.567671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.567567 2570 scope.go:117] "RemoveContainer" containerID="8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff" Jul 22 02:14:43.567671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.567562 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-kube-api-access-8xcmq" (OuterVolumeSpecName: "kube-api-access-8xcmq") pod "f95966e0-4883-4bcf-a8d2-9e92782fc0b0" (UID: "f95966e0-4883-4bcf-a8d2-9e92782fc0b0"). InnerVolumeSpecName "kube-api-access-8xcmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:14:43.567671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.567629 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f95966e0-4883-4bcf-a8d2-9e92782fc0b0" (UID: "f95966e0-4883-4bcf-a8d2-9e92782fc0b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:14:43.582937 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.582920 2570 scope.go:117] "RemoveContainer" containerID="8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff" Jul 22 02:14:43.583206 ip-10-0-130-174 kubenswrapper[2570]: E0722 02:14:43.583188 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff\": container with ID starting with 8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff not found: ID does not exist" containerID="8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff" Jul 22 02:14:43.583274 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.583254 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff"} err="failed to get container status \"8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff\": rpc error: code = NotFound desc = could not find container \"8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff\": container with ID starting with 8281a080cb1607526b5cafb9ef1e455854acd1b52fde3ffd6b227761c4e1b8ff not found: ID does not exist" Jul 22 02:14:43.666800 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.666764 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:43.666800 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.666794 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:43.666800 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.666805 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xcmq\" (UniqueName: \"kubernetes.io/projected/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-kube-api-access-8xcmq\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:43.667029 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.666817 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95966e0-4883-4bcf-a8d2-9e92782fc0b0-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:43.885190 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.885157 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84"] Jul 22 02:14:43.889060 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:43.889034 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b0ac34/test-model-b0ac34-kserve-69c6544f55-f5d84"] Jul 22 02:14:44.219578 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.219458 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" podUID="36ee45f8-6de5-405f-bf30-e496fa78f764" containerName="main" containerID="cri-o://f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305" gracePeriod=45 Jul 22 02:14:44.442991 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.442954 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:44.472305 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.472251 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:14:44.572953 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.572922 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-model-cache\") pod \"36ee45f8-6de5-405f-bf30-e496fa78f764\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " Jul 22 02:14:44.573157 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.572965 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-tmp-dir\") pod \"36ee45f8-6de5-405f-bf30-e496fa78f764\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " Jul 22 02:14:44.573157 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.572998 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8l7h\" (UniqueName: \"kubernetes.io/projected/36ee45f8-6de5-405f-bf30-e496fa78f764-kube-api-access-j8l7h\") pod \"36ee45f8-6de5-405f-bf30-e496fa78f764\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " Jul 22 02:14:44.573157 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573028 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ee45f8-6de5-405f-bf30-e496fa78f764-tls-certs\") pod \"36ee45f8-6de5-405f-bf30-e496fa78f764\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " Jul 22 02:14:44.573157 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573065 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-home\") pod \"36ee45f8-6de5-405f-bf30-e496fa78f764\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " Jul 22 02:14:44.573157 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573107 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-dshm\") pod \"36ee45f8-6de5-405f-bf30-e496fa78f764\" (UID: \"36ee45f8-6de5-405f-bf30-e496fa78f764\") " Jul 22 02:14:44.573465 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573236 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-model-cache" (OuterVolumeSpecName: "model-cache") pod "36ee45f8-6de5-405f-bf30-e496fa78f764" (UID: "36ee45f8-6de5-405f-bf30-e496fa78f764"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:14:44.573465 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573283 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36ee45f8-6de5-405f-bf30-e496fa78f764" (UID: "36ee45f8-6de5-405f-bf30-e496fa78f764"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:14:44.573465 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573372 2570 generic.go:356] "Generic (PLEG): container finished" podID="36ee45f8-6de5-405f-bf30-e496fa78f764" containerID="f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305" exitCode=0 Jul 22 02:14:44.573465 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573451 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:44.573659 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573495 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" event={"ID":"36ee45f8-6de5-405f-bf30-e496fa78f764","Type":"ContainerDied","Data":"f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305"} Jul 22 02:14:44.573659 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573495 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:44.573659 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573531 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" Jul 22 02:14:44.573659 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573551 2570 scope.go:117] "RemoveContainer" containerID="f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305" Jul 22 02:14:44.573659 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573537 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz" event={"ID":"36ee45f8-6de5-405f-bf30-e496fa78f764","Type":"ContainerDied","Data":"50893604e4d58915634500a583b1a38dde7fb0dbd18110e56eaf7b0b8951ab75"} Jul 22 02:14:44.573906 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.573825 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-home" (OuterVolumeSpecName: "home") pod "36ee45f8-6de5-405f-bf30-e496fa78f764" (UID: "36ee45f8-6de5-405f-bf30-e496fa78f764"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:14:44.575591 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.575567 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ee45f8-6de5-405f-bf30-e496fa78f764-kube-api-access-j8l7h" (OuterVolumeSpecName: "kube-api-access-j8l7h") pod "36ee45f8-6de5-405f-bf30-e496fa78f764" (UID: "36ee45f8-6de5-405f-bf30-e496fa78f764"). InnerVolumeSpecName "kube-api-access-j8l7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:14:44.575699 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.575643 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ee45f8-6de5-405f-bf30-e496fa78f764-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36ee45f8-6de5-405f-bf30-e496fa78f764" (UID: "36ee45f8-6de5-405f-bf30-e496fa78f764"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:14:44.575766 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.575746 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-dshm" (OuterVolumeSpecName: "dshm") pod "36ee45f8-6de5-405f-bf30-e496fa78f764" (UID: "36ee45f8-6de5-405f-bf30-e496fa78f764"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:14:44.590147 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.590125 2570 scope.go:117] "RemoveContainer" containerID="f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305" Jul 22 02:14:44.590430 ip-10-0-130-174 kubenswrapper[2570]: E0722 02:14:44.590411 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305\": container with ID starting with f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305 not found: ID does not exist" containerID="f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305" Jul 22 02:14:44.590492 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.590439 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305"} err="failed to get container status \"f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305\": rpc error: code = NotFound desc = could not find container \"f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305\": container with ID starting with f731ec04dc0bfdebaed3fb36a69f5003e00dd765b85ada893bdd686c6c388305 not found: ID does not exist" Jul 22 02:14:44.674588 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.674538 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8l7h\" (UniqueName: \"kubernetes.io/projected/36ee45f8-6de5-405f-bf30-e496fa78f764-kube-api-access-j8l7h\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:44.674588 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.674583 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ee45f8-6de5-405f-bf30-e496fa78f764-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:44.674588 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.674595 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:44.674853 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.674606 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ee45f8-6de5-405f-bf30-e496fa78f764-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:14:44.895806 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.895776 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz"] Jul 22 02:14:44.900541 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.900519 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1ad649/test-model-1ad649-kserve-775877555f-8zcwz"] Jul 22 02:14:44.900900 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:44.900880 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:45.598301 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:45.598273 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ee45f8-6de5-405f-bf30-e496fa78f764" path="/var/lib/kubelet/pods/36ee45f8-6de5-405f-bf30-e496fa78f764/volumes" Jul 22 02:14:45.598680 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:45.598668 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f95966e0-4883-4bcf-a8d2-9e92782fc0b0" path="/var/lib/kubelet/pods/f95966e0-4883-4bcf-a8d2-9e92782fc0b0/volumes" Jul 22 02:14:45.916061 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:45.916027 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:14:51.473393 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:51.473362 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:51.587392 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:51.587356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:51.791001 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:51.790916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:52.029249 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:52.029202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:52.812554 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:52.812521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:52.907200 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:14:52.907164 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:02.902813 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:02.902771 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:05.735707 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:05.735672 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:15:07.300720 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:07.300681 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:07.733191 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:07.733149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:14.099131 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:14.099095 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:14.172669 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:14.172631 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:14.407792 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:14.407698 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:14.654830 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:14.654801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:15.062683 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:15.062651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:15.411087 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:15.411057 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:15.539986 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:15.539955 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:18.772024 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:18.771977 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:18.878195 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:18.878154 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:18.982471 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:18.982431 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:19.586866 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:19.586835 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:19.679156 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:19.679125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:23.341866 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:23.341836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:23.603163 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:23.603134 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:15:23.868042 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:23.867961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:24.224774 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:24.224696 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:24.452004 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:24.451970 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:27.001180 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:27.001116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:27.154282 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:27.154204 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:27.330097 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:27.330001 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:27.640364 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:27.640329 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:28.219479 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:28.219445 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:28.315468 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:28.315420 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:31.415023 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:31.414992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:36.293076 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:36.293034 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:15:40.551906 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:40.551868 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:40.653801 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:40.653770 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:50.520854 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:50.520816 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:50.621619 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:50.621583 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:55.836355 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:15:55.836316 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:00.068100 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:00.067995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:00.524126 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:00.524089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:01.804965 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:01.804931 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:16:06.536078 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:06.536044 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:06.631277 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:06.631240 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:06.891918 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:06.891885 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:07.184522 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:07.184437 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:22.979011 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:22.978979 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:27.096124 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:27.096080 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:16:27.426128 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:27.426088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:27.893860 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:27.893808 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:33.928167 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:33.928133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:34.029946 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:34.029907 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:34.282371 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:34.282278 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:34.559880 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:34.559793 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:34.962353 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:34.962306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:35.303685 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:35.303599 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:35.535027 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:35.534994 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:38.284408 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:38.284376 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:38.380637 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:38.380599 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:38.478546 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:38.478513 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:38.592953 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:38.592857 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:38.986910 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:38.986872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:39.089831 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:39.089794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:42.963367 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:42.963338 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:43.333143 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:43.333067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:43.717059 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:43.717023 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:43.831830 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:43.831799 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:45.793995 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:45.793966 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:16:47.363111 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:47.363077 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:47.486813 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:47.486779 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:47.641047 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:47.640959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:47.728898 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:47.728859 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:48.051526 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:48.051438 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:48.225251 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:48.225202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:51.891165 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:51.891124 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:57.092322 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:16:57.092281 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:17:00.003372 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:00.003336 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:00.101834 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:00.101804 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:00.808161 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:00.808120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:00.905776 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:00.905745 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:01.725923 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:01.725890 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:01.827838 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:01.827804 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:05.960998 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:05.960973 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:17:05.963113 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:05.963092 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:17:09.016404 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:09.016374 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:09.113828 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:09.113792 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:13.395979 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.395949 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s"] Jul 22 02:17:13.396385 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.396254 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ee45f8-6de5-405f-bf30-e496fa78f764" containerName="main" Jul 22 02:17:13.396385 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.396269 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ee45f8-6de5-405f-bf30-e496fa78f764" containerName="main" Jul 22 02:17:13.396385 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.396281 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f95966e0-4883-4bcf-a8d2-9e92782fc0b0" containerName="main" Jul 22 02:17:13.396385 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.396287 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f95966e0-4883-4bcf-a8d2-9e92782fc0b0" containerName="main" Jul 22 02:17:13.396385 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.396332 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f95966e0-4883-4bcf-a8d2-9e92782fc0b0" containerName="main" Jul 22 02:17:13.396385 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.396346 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ee45f8-6de5-405f-bf30-e496fa78f764" containerName="main" Jul 22 02:17:13.399079 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.399061 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.402026 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.402007 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"default-dockercfg-cgpqq\"" Jul 22 02:17:13.402132 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.402008 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"openshift-service-ca.crt\"" Jul 22 02:17:13.402852 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.402834 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"kube-root-ca.crt\"" Jul 22 02:17:13.402937 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.402920 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"auth-test-model-eba9c8-kserve-self-signed-certs\"" Jul 22 02:17:13.411002 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.410976 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s"] Jul 22 02:17:13.501460 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.501424 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:13.566196 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.566160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tls-certs\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.566196 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.566198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tstnb\" (UniqueName: \"kubernetes.io/projected/31ddf0f7-9805-436f-a1d8-603dd4fc1891-kube-api-access-tstnb\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.566436 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.566278 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tmp-dir\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.566436 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.566326 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-home\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.566436 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.566356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-dshm\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.566545 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.566447 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-model-cache\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.667885 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.667801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-model-cache\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.667885 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.667847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tls-certs\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.667885 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.667872 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstnb\" (UniqueName: \"kubernetes.io/projected/31ddf0f7-9805-436f-a1d8-603dd4fc1891-kube-api-access-tstnb\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.668161 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.667910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tmp-dir\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.668161 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.667950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-home\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.668161 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.667986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-dshm\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.668358 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.668208 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-model-cache\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.668358 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.668338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tmp-dir\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.668461 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.668442 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-home\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.670169 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.670150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-dshm\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.670482 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.670461 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tls-certs\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.680900 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.680874 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstnb\" (UniqueName: \"kubernetes.io/projected/31ddf0f7-9805-436f-a1d8-603dd4fc1891-kube-api-access-tstnb\") pod \"auth-test-model-eba9c8-kserve-58947f88cf-88h6s\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.707946 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.707905 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:13.837441 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:13.837408 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s"] Jul 22 02:17:13.840109 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:17:13.840082 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31ddf0f7_9805_436f_a1d8_603dd4fc1891.slice/crio-b8335fc856769a9ed51de98458d49f57b61fe79261b679d206758cadc2405cbb WatchSource:0}: Error finding container b8335fc856769a9ed51de98458d49f57b61fe79261b679d206758cadc2405cbb: Status 404 returned error can't find the container with id b8335fc856769a9ed51de98458d49f57b61fe79261b679d206758cadc2405cbb Jul 22 02:17:14.167521 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:14.167484 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" event={"ID":"31ddf0f7-9805-436f-a1d8-603dd4fc1891","Type":"ContainerStarted","Data":"0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf"} Jul 22 02:17:14.167521 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:14.167526 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" event={"ID":"31ddf0f7-9805-436f-a1d8-603dd4fc1891","Type":"ContainerStarted","Data":"b8335fc856769a9ed51de98458d49f57b61fe79261b679d206758cadc2405cbb"} Jul 22 02:17:14.185834 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:14.185787 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" podStartSLOduration=1.185772466 podStartE2EDuration="1.185772466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:17:14.184609523 +0000 UTC m=+2408.682050930" watchObservedRunningTime="2026-07-22 02:17:14.185772466 +0000 UTC m=+2408.683213822" Jul 22 02:17:16.594179 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:16.594149 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:17:23.708068 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:23.708030 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:23.708551 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:23.708093 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:23.720592 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:23.720570 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:24.226420 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:24.226393 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:17:40.499682 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.499600 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn"] Jul 22 02:17:40.502990 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.502970 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.505354 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.505333 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"kube-root-ca.crt\"" Jul 22 02:17:40.505488 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.505399 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"openshift-service-ca.crt\"" Jul 22 02:17:40.506415 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.506397 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"auth-test-model-c6960c-kserve-self-signed-certs\"" Jul 22 02:17:40.506484 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.506458 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"default-dockercfg-97vlp\"" Jul 22 02:17:40.513788 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.513766 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn"] Jul 22 02:17:40.540598 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.540560 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:40.571738 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.571697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-tmp-dir\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.571926 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.571745 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/45baf74e-acab-4a6d-9267-f1de9005135e-kube-api-access-bgtgj\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.571926 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.571810 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-home\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.571926 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.571838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45baf74e-acab-4a6d-9267-f1de9005135e-tls-certs\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.571926 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.571863 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-model-cache\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.571926 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.571889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-dshm\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.673440 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.672800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-tmp-dir\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.673440 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.672846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/45baf74e-acab-4a6d-9267-f1de9005135e-kube-api-access-bgtgj\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.673440 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.672885 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-home\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.673440 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.672910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45baf74e-acab-4a6d-9267-f1de9005135e-tls-certs\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.673440 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.672935 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-model-cache\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.673440 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.672960 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-dshm\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.674230 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.674186 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-home\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.674603 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.674562 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-model-cache\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.674722 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.674609 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-tmp-dir\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.676287 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.676265 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-dshm\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.676917 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.676895 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45baf74e-acab-4a6d-9267-f1de9005135e-tls-certs\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.683644 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.683620 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/45baf74e-acab-4a6d-9267-f1de9005135e-kube-api-access-bgtgj\") pod \"auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.812784 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.812690 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:40.952109 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:40.952074 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn"] Jul 22 02:17:40.955007 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:17:40.954977 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45baf74e_acab_4a6d_9267_f1de9005135e.slice/crio-829ba73663169a01087d118b06069238eb031021cc7928aa0997336f8e1b1386 WatchSource:0}: Error finding container 829ba73663169a01087d118b06069238eb031021cc7928aa0997336f8e1b1386: Status 404 returned error can't find the container with id 829ba73663169a01087d118b06069238eb031021cc7928aa0997336f8e1b1386 Jul 22 02:17:41.286085 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:41.286046 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" event={"ID":"45baf74e-acab-4a6d-9267-f1de9005135e","Type":"ContainerStarted","Data":"553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723"} Jul 22 02:17:41.286307 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:41.286095 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" event={"ID":"45baf74e-acab-4a6d-9267-f1de9005135e","Type":"ContainerStarted","Data":"829ba73663169a01087d118b06069238eb031021cc7928aa0997336f8e1b1386"} Jul 22 02:17:41.305339 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:41.305286 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" podStartSLOduration=1.305270862 podStartE2EDuration="1.305270862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:17:41.304509057 +0000 UTC m=+2435.801950415" watchObservedRunningTime="2026-07-22 02:17:41.305270862 +0000 UTC m=+2435.802712218" Jul 22 02:17:43.992798 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:43.992760 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:17:47.484587 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:47.484538 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:47.581031 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:47.580995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:50.813804 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:50.813766 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:50.813804 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:50.813816 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:50.826234 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:50.826192 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:17:51.342379 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:17:51.342347 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:18:00.498779 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:18:00.498741 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:18:16.404161 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:18:16.404123 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:18:26.898847 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:18:26.898806 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:35.895229 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:18:35.895176 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:18:56.211661 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:18:56.211621 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:19:16.103147 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:16.103062 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:19:35.001160 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:35.001127 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:19:42.141760 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:42.141723 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:49.796697 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:49.796663 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s"] Jul 22 02:19:49.810737 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:49.810708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:50.596165 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:50.596134 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn"] Jul 22 02:19:50.615973 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:50.615942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:54.087047 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.087013 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:19:54.692898 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.692862 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz"] Jul 22 02:19:54.696027 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.696009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.698293 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.698273 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"sub-test-model-eba9c8-kserve-self-signed-certs\"" Jul 22 02:19:54.704421 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.704398 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz"] Jul 22 02:19:54.719223 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.719184 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:54.796647 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.796606 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-home\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.796811 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.796655 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-model-cache\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.796811 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.796683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcn5f\" (UniqueName: \"kubernetes.io/projected/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-kube-api-access-rcn5f\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.796811 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.796728 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-dshm\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.796811 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.796768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tls-certs\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.796811 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.796788 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tmp-dir\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.897688 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.897626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tls-certs\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.897688 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.897691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tmp-dir\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.897946 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.897733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-home\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.897946 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.897766 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-model-cache\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.897946 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.897883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcn5f\" (UniqueName: \"kubernetes.io/projected/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-kube-api-access-rcn5f\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.897946 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.897916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-dshm\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.898186 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.898153 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tmp-dir\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.898286 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.898186 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-model-cache\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.898286 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.898268 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-home\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.900132 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.900109 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-dshm\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.900364 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.900346 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tls-certs\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:54.906868 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:54.906839 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcn5f\" (UniqueName: \"kubernetes.io/projected/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-kube-api-access-rcn5f\") pod \"sub-test-model-eba9c8-kserve-5c66bd4747-rgttz\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:55.006013 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:55.005924 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:19:55.135025 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:55.134956 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz"] Jul 22 02:19:55.137121 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:19:55.137093 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf672c8b_bb3b_4816_a6c8_15d9c1af7ab5.slice/crio-a36f7517cc38256212a6e5b494a3f64e4f7ff1d227f86d7400f62fbf9c300f79 WatchSource:0}: Error finding container a36f7517cc38256212a6e5b494a3f64e4f7ff1d227f86d7400f62fbf9c300f79: Status 404 returned error can't find the container with id a36f7517cc38256212a6e5b494a3f64e4f7ff1d227f86d7400f62fbf9c300f79 Jul 22 02:19:55.846561 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:55.846527 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" event={"ID":"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5","Type":"ContainerStarted","Data":"1a9285cc4db5de3c79ab6782eb9de47f30ae8c7beb2f121bca1e6138cbfc7eb7"} Jul 22 02:19:55.846727 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:55.846572 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" event={"ID":"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5","Type":"ContainerStarted","Data":"a36f7517cc38256212a6e5b494a3f64e4f7ff1d227f86d7400f62fbf9c300f79"} Jul 22 02:19:55.865271 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:19:55.865205 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" podStartSLOduration=1.865188468 podStartE2EDuration="1.865188468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:19:55.863801531 +0000 UTC m=+2570.361242888" watchObservedRunningTime="2026-07-22 02:19:55.865188468 +0000 UTC m=+2570.362629824" Jul 22 02:20:04.838267 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:04.838183 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" podUID="31ddf0f7-9805-436f-a1d8-603dd4fc1891" containerName="main" containerID="cri-o://0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf" gracePeriod=45 Jul 22 02:20:05.006363 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.006333 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:20:05.006466 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.006402 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:20:05.024670 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.024636 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:20:05.087327 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.087302 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:20:05.185660 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.185621 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-model-cache\") pod \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " Jul 22 02:20:05.185660 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.185667 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tmp-dir\") pod \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " Jul 22 02:20:05.185980 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.185701 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tls-certs\") pod \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " Jul 22 02:20:05.185980 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.185753 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-dshm\") pod \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " Jul 22 02:20:05.185980 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.185780 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tstnb\" (UniqueName: \"kubernetes.io/projected/31ddf0f7-9805-436f-a1d8-603dd4fc1891-kube-api-access-tstnb\") pod \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " Jul 22 02:20:05.185980 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.185852 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-home\") pod \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\" (UID: \"31ddf0f7-9805-436f-a1d8-603dd4fc1891\") " Jul 22 02:20:05.185980 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.185933 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-model-cache" (OuterVolumeSpecName: "model-cache") pod "31ddf0f7-9805-436f-a1d8-603dd4fc1891" (UID: "31ddf0f7-9805-436f-a1d8-603dd4fc1891"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:20:05.186193 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.186001 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31ddf0f7-9805-436f-a1d8-603dd4fc1891" (UID: "31ddf0f7-9805-436f-a1d8-603dd4fc1891"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:20:05.186193 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.186153 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:05.186193 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.186160 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-home" (OuterVolumeSpecName: "home") pod "31ddf0f7-9805-436f-a1d8-603dd4fc1891" (UID: "31ddf0f7-9805-436f-a1d8-603dd4fc1891"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:20:05.186193 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.186173 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:05.188146 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.188123 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-dshm" (OuterVolumeSpecName: "dshm") pod "31ddf0f7-9805-436f-a1d8-603dd4fc1891" (UID: "31ddf0f7-9805-436f-a1d8-603dd4fc1891"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:20:05.188286 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.188128 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31ddf0f7-9805-436f-a1d8-603dd4fc1891" (UID: "31ddf0f7-9805-436f-a1d8-603dd4fc1891"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:20:05.188286 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.188151 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ddf0f7-9805-436f-a1d8-603dd4fc1891-kube-api-access-tstnb" (OuterVolumeSpecName: "kube-api-access-tstnb") pod "31ddf0f7-9805-436f-a1d8-603dd4fc1891" (UID: "31ddf0f7-9805-436f-a1d8-603dd4fc1891"). InnerVolumeSpecName "kube-api-access-tstnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:20:05.287013 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.286978 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ddf0f7-9805-436f-a1d8-603dd4fc1891-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:05.287013 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.287006 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:05.287013 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.287015 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tstnb\" (UniqueName: \"kubernetes.io/projected/31ddf0f7-9805-436f-a1d8-603dd4fc1891-kube-api-access-tstnb\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:05.287013 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.287023 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ddf0f7-9805-436f-a1d8-603dd4fc1891-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:05.616283 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.616244 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" podUID="45baf74e-acab-4a6d-9267-f1de9005135e" containerName="main" containerID="cri-o://553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723" gracePeriod=45 Jul 22 02:20:05.864571 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.864547 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:20:05.888578 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.888543 2570 generic.go:356] "Generic (PLEG): container finished" podID="31ddf0f7-9805-436f-a1d8-603dd4fc1891" containerID="0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf" exitCode=0 Jul 22 02:20:05.888578 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.888582 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" Jul 22 02:20:05.888822 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.888619 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" event={"ID":"31ddf0f7-9805-436f-a1d8-603dd4fc1891","Type":"ContainerDied","Data":"0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf"} Jul 22 02:20:05.888822 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.888669 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s" event={"ID":"31ddf0f7-9805-436f-a1d8-603dd4fc1891","Type":"ContainerDied","Data":"b8335fc856769a9ed51de98458d49f57b61fe79261b679d206758cadc2405cbb"} Jul 22 02:20:05.888822 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.888693 2570 scope.go:117] "RemoveContainer" containerID="0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf" Jul 22 02:20:05.890116 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.890089 2570 generic.go:356] "Generic (PLEG): container finished" podID="45baf74e-acab-4a6d-9267-f1de9005135e" containerID="553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723" exitCode=0 Jul 22 02:20:05.890437 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.890421 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" Jul 22 02:20:05.890526 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.890502 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" event={"ID":"45baf74e-acab-4a6d-9267-f1de9005135e","Type":"ContainerDied","Data":"553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723"} Jul 22 02:20:05.890577 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.890546 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn" event={"ID":"45baf74e-acab-4a6d-9267-f1de9005135e","Type":"ContainerDied","Data":"829ba73663169a01087d118b06069238eb031021cc7928aa0997336f8e1b1386"} Jul 22 02:20:05.898646 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.898627 2570 scope.go:117] "RemoveContainer" containerID="0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf" Jul 22 02:20:05.898956 ip-10-0-130-174 kubenswrapper[2570]: E0722 02:20:05.898929 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf\": container with ID starting with 0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf not found: ID does not exist" containerID="0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf" Jul 22 02:20:05.899049 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.898976 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf"} err="failed to get container status \"0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf\": rpc error: code = NotFound desc = could not find container \"0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf\": container with ID starting with 0596a573394bad06dc7104bc5ed7764e62afe13a873c46cd3d3b62ee0926ebcf not found: ID does not exist" Jul 22 02:20:05.899049 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.899000 2570 scope.go:117] "RemoveContainer" containerID="553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723" Jul 22 02:20:05.903714 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.903666 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:20:05.905405 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.905382 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s"] Jul 22 02:20:05.907308 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.907290 2570 scope.go:117] "RemoveContainer" containerID="553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723" Jul 22 02:20:05.907581 ip-10-0-130-174 kubenswrapper[2570]: E0722 02:20:05.907563 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723\": container with ID starting with 553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723 not found: ID does not exist" containerID="553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723" Jul 22 02:20:05.907642 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.907592 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723"} err="failed to get container status \"553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723\": rpc error: code = NotFound desc = could not find container \"553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723\": container with ID starting with 553f0bc850e121d8c1061373b8dbbfba9e703ecf7909495f51d0eb10ab08a723 not found: ID does not exist" Jul 22 02:20:05.916317 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.912089 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/auth-test-model-eba9c8-kserve-58947f88cf-88h6s"] Jul 22 02:20:05.992171 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992132 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-dshm\") pod \"45baf74e-acab-4a6d-9267-f1de9005135e\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " Jul 22 02:20:05.992416 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992275 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45baf74e-acab-4a6d-9267-f1de9005135e-tls-certs\") pod \"45baf74e-acab-4a6d-9267-f1de9005135e\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " Jul 22 02:20:05.992416 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992311 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-tmp-dir\") pod \"45baf74e-acab-4a6d-9267-f1de9005135e\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " Jul 22 02:20:05.992416 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992352 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/45baf74e-acab-4a6d-9267-f1de9005135e-kube-api-access-bgtgj\") pod \"45baf74e-acab-4a6d-9267-f1de9005135e\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " Jul 22 02:20:05.992416 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992377 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-home\") pod \"45baf74e-acab-4a6d-9267-f1de9005135e\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " Jul 22 02:20:05.992416 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992399 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-model-cache\") pod \"45baf74e-acab-4a6d-9267-f1de9005135e\" (UID: \"45baf74e-acab-4a6d-9267-f1de9005135e\") " Jul 22 02:20:05.992699 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992592 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45baf74e-acab-4a6d-9267-f1de9005135e" (UID: "45baf74e-acab-4a6d-9267-f1de9005135e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:20:05.992699 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992640 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-home" (OuterVolumeSpecName: "home") pod "45baf74e-acab-4a6d-9267-f1de9005135e" (UID: "45baf74e-acab-4a6d-9267-f1de9005135e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:20:05.992699 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992683 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:05.992699 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.992690 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-model-cache" (OuterVolumeSpecName: "model-cache") pod "45baf74e-acab-4a6d-9267-f1de9005135e" (UID: "45baf74e-acab-4a6d-9267-f1de9005135e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:20:05.994516 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.994484 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-dshm" (OuterVolumeSpecName: "dshm") pod "45baf74e-acab-4a6d-9267-f1de9005135e" (UID: "45baf74e-acab-4a6d-9267-f1de9005135e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:20:05.994631 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.994529 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45baf74e-acab-4a6d-9267-f1de9005135e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45baf74e-acab-4a6d-9267-f1de9005135e" (UID: "45baf74e-acab-4a6d-9267-f1de9005135e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:20:05.994631 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:05.994538 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45baf74e-acab-4a6d-9267-f1de9005135e-kube-api-access-bgtgj" (OuterVolumeSpecName: "kube-api-access-bgtgj") pod "45baf74e-acab-4a6d-9267-f1de9005135e" (UID: "45baf74e-acab-4a6d-9267-f1de9005135e"). InnerVolumeSpecName "kube-api-access-bgtgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:20:06.093594 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:06.093555 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:06.093594 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:06.093587 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45baf74e-acab-4a6d-9267-f1de9005135e-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:06.093594 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:06.093599 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/45baf74e-acab-4a6d-9267-f1de9005135e-kube-api-access-bgtgj\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:06.093816 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:06.093608 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:06.093816 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:06.093617 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45baf74e-acab-4a6d-9267-f1de9005135e-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:20:06.210878 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:06.210846 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn"] Jul 22 02:20:06.214468 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:06.214444 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/auth-test-model-c6960c-kserve-6b45b6bd47-f5vkn"] Jul 22 02:20:07.390235 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:07.390168 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:20:07.597948 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:07.597920 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ddf0f7-9805-436f-a1d8-603dd4fc1891" path="/var/lib/kubelet/pods/31ddf0f7-9805-436f-a1d8-603dd4fc1891/volumes" Jul 22 02:20:07.598303 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:07.598290 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45baf74e-acab-4a6d-9267-f1de9005135e" path="/var/lib/kubelet/pods/45baf74e-acab-4a6d-9267-f1de9005135e/volumes" Jul 22 02:20:15.194693 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.194658 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv"] Jul 22 02:20:15.195071 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.194951 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45baf74e-acab-4a6d-9267-f1de9005135e" containerName="main" Jul 22 02:20:15.195071 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.194963 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="45baf74e-acab-4a6d-9267-f1de9005135e" containerName="main" Jul 22 02:20:15.195071 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.194980 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ddf0f7-9805-436f-a1d8-603dd4fc1891" containerName="main" Jul 22 02:20:15.195071 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.194986 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ddf0f7-9805-436f-a1d8-603dd4fc1891" containerName="main" Jul 22 02:20:15.195071 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.195033 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="45baf74e-acab-4a6d-9267-f1de9005135e" containerName="main" Jul 22 02:20:15.195071 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.195047 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ddf0f7-9805-436f-a1d8-603dd4fc1891" containerName="main" Jul 22 02:20:15.198018 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.198002 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.200376 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.200355 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"kube-root-ca.crt\"" Jul 22 02:20:15.201293 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.201276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"default-dockercfg-97vlp\"" Jul 22 02:20:15.201358 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.201292 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"openshift-service-ca.crt\"" Jul 22 02:20:15.201358 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.201276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"sub-test-model-c6960c-kserve-self-signed-certs\"" Jul 22 02:20:15.208878 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.208855 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv"] Jul 22 02:20:15.223524 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.223495 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:20:15.370282 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.370246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tmp-dir\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.370282 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.370285 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tls-certs\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.370496 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.370353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-model-cache\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.370496 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.370371 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-dshm\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.370496 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.370395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-home\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.370496 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.370417 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p44w\" (UniqueName: \"kubernetes.io/projected/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-kube-api-access-2p44w\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471347 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tmp-dir\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471347 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tls-certs\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471347 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471347 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-model-cache\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471596 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471366 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-dshm\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471596 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471388 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-home\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471596 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p44w\" (UniqueName: \"kubernetes.io/projected/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-kube-api-access-2p44w\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471753 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471676 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tmp-dir\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471753 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471741 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-model-cache\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.471865 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.471760 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-home\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.473723 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.473703 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-dshm\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.474017 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.473995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tls-certs\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.480748 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.480726 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p44w\" (UniqueName: \"kubernetes.io/projected/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-kube-api-access-2p44w\") pod \"sub-test-model-c6960c-kserve-6474cf46bf-4wktv\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.506574 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.506548 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:15.634298 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.634271 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv"] Jul 22 02:20:15.636526 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:20:15.636492 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbd4c9ef_703e_4c84_925f_fadb2e0da48b.slice/crio-90ca337150e2dad68a80005dfc88ae8c8adb3a99876c7c0ff2372837e2364cc2 WatchSource:0}: Error finding container 90ca337150e2dad68a80005dfc88ae8c8adb3a99876c7c0ff2372837e2364cc2: Status 404 returned error can't find the container with id 90ca337150e2dad68a80005dfc88ae8c8adb3a99876c7c0ff2372837e2364cc2 Jul 22 02:20:15.935397 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.935363 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" event={"ID":"dbd4c9ef-703e-4c84-925f-fadb2e0da48b","Type":"ContainerStarted","Data":"456ec35c22bfd5f0603e3771d3f3278879cea843356a5322e8f92a19824ccf51"} Jul 22 02:20:15.935397 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.935406 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" event={"ID":"dbd4c9ef-703e-4c84-925f-fadb2e0da48b","Type":"ContainerStarted","Data":"90ca337150e2dad68a80005dfc88ae8c8adb3a99876c7c0ff2372837e2364cc2"} Jul 22 02:20:15.961028 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:15.960973 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" podStartSLOduration=0.960957733 podStartE2EDuration="960.957733ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:20:15.959165476 +0000 UTC m=+2590.456606855" watchObservedRunningTime="2026-07-22 02:20:15.960957733 +0000 UTC m=+2590.458399090" Jul 22 02:20:19.294081 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:19.294050 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:20:25.507046 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:25.507012 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:25.507436 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:25.507349 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:25.520038 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:25.520013 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:25.987309 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:25.987277 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:20:32.896267 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:32.896171 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:20:52.799842 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:52.799801 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:20:59.003052 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:59.003010 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz"] Jul 22 02:20:59.028935 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:59.028908 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:20:59.986784 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:20:59.986749 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv"] Jul 22 02:21:00.009629 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:00.009589 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:10.525981 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:10.525939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:11.891033 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:11.890997 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:21:14.023027 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.022970 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" podUID="bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" containerName="main" containerID="cri-o://1a9285cc4db5de3c79ab6782eb9de47f30ae8c7beb2f121bca1e6138cbfc7eb7" gracePeriod=45 Jul 22 02:21:14.171189 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.171162 2570 generic.go:356] "Generic (PLEG): container finished" podID="bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" containerID="1a9285cc4db5de3c79ab6782eb9de47f30ae8c7beb2f121bca1e6138cbfc7eb7" exitCode=0 Jul 22 02:21:14.171363 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.171230 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" event={"ID":"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5","Type":"ContainerDied","Data":"1a9285cc4db5de3c79ab6782eb9de47f30ae8c7beb2f121bca1e6138cbfc7eb7"} Jul 22 02:21:14.264379 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.264355 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:21:14.367359 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367333 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tls-certs\") pod \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " Jul 22 02:21:14.367509 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367371 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-home\") pod \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " Jul 22 02:21:14.367509 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367417 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-dshm\") pod \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " Jul 22 02:21:14.367509 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367444 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcn5f\" (UniqueName: \"kubernetes.io/projected/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-kube-api-access-rcn5f\") pod \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " Jul 22 02:21:14.367509 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367467 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-model-cache\") pod \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " Jul 22 02:21:14.367509 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367500 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tmp-dir\") pod \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\" (UID: \"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5\") " Jul 22 02:21:14.367777 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367749 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" (UID: "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:14.367861 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367756 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-home" (OuterVolumeSpecName: "home") pod "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" (UID: "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:14.367861 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.367833 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" (UID: "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:14.369554 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.369528 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" (UID: "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:21:14.369662 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.369605 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-dshm" (OuterVolumeSpecName: "dshm") pod "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" (UID: "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:14.369662 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.369604 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-kube-api-access-rcn5f" (OuterVolumeSpecName: "kube-api-access-rcn5f") pod "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" (UID: "bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5"). InnerVolumeSpecName "kube-api-access-rcn5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:21:14.468324 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.468291 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:14.468324 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.468320 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcn5f\" (UniqueName: \"kubernetes.io/projected/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-kube-api-access-rcn5f\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:14.468324 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.468330 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:14.468549 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.468340 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:14.468549 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.468348 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:14.468549 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:14.468355 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:15.007142 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.007098 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" podUID="dbd4c9ef-703e-4c84-925f-fadb2e0da48b" containerName="main" containerID="cri-o://456ec35c22bfd5f0603e3771d3f3278879cea843356a5322e8f92a19824ccf51" gracePeriod=45 Jul 22 02:21:15.177361 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.177334 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" Jul 22 02:21:15.177768 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.177332 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz" event={"ID":"bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5","Type":"ContainerDied","Data":"a36f7517cc38256212a6e5b494a3f64e4f7ff1d227f86d7400f62fbf9c300f79"} Jul 22 02:21:15.177768 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.177468 2570 scope.go:117] "RemoveContainer" containerID="1a9285cc4db5de3c79ab6782eb9de47f30ae8c7beb2f121bca1e6138cbfc7eb7" Jul 22 02:21:15.179372 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.179333 2570 generic.go:356] "Generic (PLEG): container finished" podID="dbd4c9ef-703e-4c84-925f-fadb2e0da48b" containerID="456ec35c22bfd5f0603e3771d3f3278879cea843356a5322e8f92a19824ccf51" exitCode=0 Jul 22 02:21:15.179499 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.179403 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" event={"ID":"dbd4c9ef-703e-4c84-925f-fadb2e0da48b","Type":"ContainerDied","Data":"456ec35c22bfd5f0603e3771d3f3278879cea843356a5322e8f92a19824ccf51"} Jul 22 02:21:15.201094 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.201070 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz"] Jul 22 02:21:15.204870 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.204847 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/sub-test-model-eba9c8-kserve-5c66bd4747-rgttz"] Jul 22 02:21:15.257487 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.257432 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:21:15.376378 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376338 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-dshm\") pod \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " Jul 22 02:21:15.376378 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376381 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tmp-dir\") pod \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " Jul 22 02:21:15.376620 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376420 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tls-certs\") pod \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " Jul 22 02:21:15.376620 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376456 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p44w\" (UniqueName: \"kubernetes.io/projected/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-kube-api-access-2p44w\") pod \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " Jul 22 02:21:15.376620 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376491 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-home\") pod \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " Jul 22 02:21:15.376620 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376574 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-model-cache\") pod \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\" (UID: \"dbd4c9ef-703e-4c84-925f-fadb2e0da48b\") " Jul 22 02:21:15.376851 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376679 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbd4c9ef-703e-4c84-925f-fadb2e0da48b" (UID: "dbd4c9ef-703e-4c84-925f-fadb2e0da48b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:15.376893 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376841 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:15.376893 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376857 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-home" (OuterVolumeSpecName: "home") pod "dbd4c9ef-703e-4c84-925f-fadb2e0da48b" (UID: "dbd4c9ef-703e-4c84-925f-fadb2e0da48b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:15.376893 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.376867 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbd4c9ef-703e-4c84-925f-fadb2e0da48b" (UID: "dbd4c9ef-703e-4c84-925f-fadb2e0da48b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:15.378640 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.378612 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-dshm" (OuterVolumeSpecName: "dshm") pod "dbd4c9ef-703e-4c84-925f-fadb2e0da48b" (UID: "dbd4c9ef-703e-4c84-925f-fadb2e0da48b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:15.378747 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.378660 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-kube-api-access-2p44w" (OuterVolumeSpecName: "kube-api-access-2p44w") pod "dbd4c9ef-703e-4c84-925f-fadb2e0da48b" (UID: "dbd4c9ef-703e-4c84-925f-fadb2e0da48b"). InnerVolumeSpecName "kube-api-access-2p44w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:21:15.378747 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.378674 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbd4c9ef-703e-4c84-925f-fadb2e0da48b" (UID: "dbd4c9ef-703e-4c84-925f-fadb2e0da48b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:21:15.477388 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.477346 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:15.477388 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.477380 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:15.477388 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.477392 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:15.477645 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.477404 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p44w\" (UniqueName: \"kubernetes.io/projected/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-kube-api-access-2p44w\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:15.477645 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.477416 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd4c9ef-703e-4c84-925f-fadb2e0da48b-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:21:15.597854 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:15.597776 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" path="/var/lib/kubelet/pods/bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5/volumes" Jul 22 02:21:16.185038 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:16.185007 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" Jul 22 02:21:16.185577 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:16.185058 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv" event={"ID":"dbd4c9ef-703e-4c84-925f-fadb2e0da48b","Type":"ContainerDied","Data":"90ca337150e2dad68a80005dfc88ae8c8adb3a99876c7c0ff2372837e2364cc2"} Jul 22 02:21:16.185577 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:16.185118 2570 scope.go:117] "RemoveContainer" containerID="456ec35c22bfd5f0603e3771d3f3278879cea843356a5322e8f92a19824ccf51" Jul 22 02:21:16.203391 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:16.203364 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv"] Jul 22 02:21:16.208238 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:16.208201 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/sub-test-model-c6960c-kserve-6474cf46bf-4wktv"] Jul 22 02:21:17.598137 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:17.598104 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd4c9ef-703e-4c84-925f-fadb2e0da48b" path="/var/lib/kubelet/pods/dbd4c9ef-703e-4c84-925f-fadb2e0da48b/volumes" Jul 22 02:21:20.363371 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.363313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:20.697993 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.697922 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48"] Jul 22 02:21:20.698293 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.698279 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd4c9ef-703e-4c84-925f-fadb2e0da48b" containerName="main" Jul 22 02:21:20.698342 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.698296 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd4c9ef-703e-4c84-925f-fadb2e0da48b" containerName="main" Jul 22 02:21:20.698342 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.698305 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" containerName="main" Jul 22 02:21:20.698342 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.698311 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" containerName="main" Jul 22 02:21:20.698439 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.698358 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd4c9ef-703e-4c84-925f-fadb2e0da48b" containerName="main" Jul 22 02:21:20.698439 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.698368 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf672c8b-bb3b-4816-a6c8-15d9c1af7ab5" containerName="main" Jul 22 02:21:20.702375 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.702360 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.705769 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.705735 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"openshift-service-ca.crt\"" Jul 22 02:21:20.705910 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.705806 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"rate-test-model-eba9c8-kserve-self-signed-certs\"" Jul 22 02:21:20.705910 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.705820 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"kube-root-ca.crt\"" Jul 22 02:21:20.705910 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.705885 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eba9c8\"/\"default-dockercfg-cgpqq\"" Jul 22 02:21:20.712407 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.712385 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48"] Jul 22 02:21:20.722872 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.722848 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:20.823401 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.823363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-tmp-dir\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.823401 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.823401 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-home\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.823635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.823437 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-model-cache\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.823635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.823483 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5km\" (UniqueName: \"kubernetes.io/projected/11d4b967-185b-40d8-8286-239b512acc4a-kube-api-access-6f5km\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.823635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.823560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11d4b967-185b-40d8-8286-239b512acc4a-tls-certs\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.823635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.823601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-dshm\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.924916 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.924872 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11d4b967-185b-40d8-8286-239b512acc4a-tls-certs\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.925112 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.924922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-dshm\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.925112 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.924964 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-tmp-dir\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.925112 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.924989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-home\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.925112 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.925031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-model-cache\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.925112 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.925059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5km\" (UniqueName: \"kubernetes.io/projected/11d4b967-185b-40d8-8286-239b512acc4a-kube-api-access-6f5km\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.925465 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.925447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-tmp-dir\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.925547 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.925496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-home\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.925547 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.925509 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-model-cache\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.927287 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.927264 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-dshm\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.927452 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.927435 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11d4b967-185b-40d8-8286-239b512acc4a-tls-certs\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:20.934728 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:20.934702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5km\" (UniqueName: \"kubernetes.io/projected/11d4b967-185b-40d8-8286-239b512acc4a-kube-api-access-6f5km\") pod \"rate-test-model-eba9c8-kserve-5896d6d96c-ldf48\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:21.011635 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:21.011551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:21.137024 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:21.137000 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48"] Jul 22 02:21:21.139064 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:21:21.139034 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11d4b967_185b_40d8_8286_239b512acc4a.slice/crio-9e8d6cf9e3256bbd54c7771362da7aea6989d7268c0e5c35f00c1cb0da6dda54 WatchSource:0}: Error finding container 9e8d6cf9e3256bbd54c7771362da7aea6989d7268c0e5c35f00c1cb0da6dda54: Status 404 returned error can't find the container with id 9e8d6cf9e3256bbd54c7771362da7aea6989d7268c0e5c35f00c1cb0da6dda54 Jul 22 02:21:21.206866 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:21.206836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" event={"ID":"11d4b967-185b-40d8-8286-239b512acc4a","Type":"ContainerStarted","Data":"d7d67f37e18faf4c720729ebd63651daabb09665a2a754a7e652cbe1310d3d66"} Jul 22 02:21:21.206974 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:21.206881 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" event={"ID":"11d4b967-185b-40d8-8286-239b512acc4a","Type":"ContainerStarted","Data":"9e8d6cf9e3256bbd54c7771362da7aea6989d7268c0e5c35f00c1cb0da6dda54"} Jul 22 02:21:21.224934 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:21.224888 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" podStartSLOduration=1.224873543 podStartE2EDuration="1.224873543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:21.223205239 +0000 UTC m=+2655.720646610" watchObservedRunningTime="2026-07-22 02:21:21.224873543 +0000 UTC m=+2655.722314899" Jul 22 02:21:24.597409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:24.597374 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:21:31.012179 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:31.012140 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:31.012711 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:31.012372 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:31.024789 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:31.024759 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:31.258841 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:31.258808 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:21:48.696276 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.696242 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72"] Jul 22 02:21:48.699378 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.699362 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.702492 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.702472 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"kube-root-ca.crt\"" Jul 22 02:21:48.702623 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.702497 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"openshift-service-ca.crt\"" Jul 22 02:21:48.702623 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.702519 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"rate-test-model-c6960c-kserve-self-signed-certs\"" Jul 22 02:21:48.702623 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.702522 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6960c\"/\"default-dockercfg-97vlp\"" Jul 22 02:21:48.708069 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.708049 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72"] Jul 22 02:21:48.719739 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.719715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:48.733309 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.733283 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2702c7fa-af5a-4454-a7eb-d04b25790a61-tls-certs\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.733405 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.733321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-model-cache\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.733405 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.733344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tpm\" (UniqueName: \"kubernetes.io/projected/2702c7fa-af5a-4454-a7eb-d04b25790a61-kube-api-access-v2tpm\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.733496 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.733412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-tmp-dir\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.733496 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.733450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-home\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.733496 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.733475 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-dshm\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.833949 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.833919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2702c7fa-af5a-4454-a7eb-d04b25790a61-tls-certs\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.834109 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.833955 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-model-cache\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.834109 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.833979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tpm\" (UniqueName: \"kubernetes.io/projected/2702c7fa-af5a-4454-a7eb-d04b25790a61-kube-api-access-v2tpm\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.834109 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.834004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-tmp-dir\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.834109 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.834024 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-home\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.834109 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.834047 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-dshm\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.834390 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.834347 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-model-cache\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.834427 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.834413 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-tmp-dir\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.834461 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.834447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-home\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.836292 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.836272 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-dshm\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.836542 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.836525 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2702c7fa-af5a-4454-a7eb-d04b25790a61-tls-certs\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:48.842526 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:48.842505 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tpm\" (UniqueName: \"kubernetes.io/projected/2702c7fa-af5a-4454-a7eb-d04b25790a61-kube-api-access-v2tpm\") pod \"rate-test-model-c6960c-kserve-55bdf76547-msq72\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:49.008941 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:49.008856 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:49.135584 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:49.135548 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72"] Jul 22 02:21:49.139119 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:21:49.139090 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2702c7fa_af5a_4454_a7eb_d04b25790a61.slice/crio-2727badfe5fd7d8633b6fb610d49d0f4cc416f3fbaf3c2c6321ebeabecc5c2f7 WatchSource:0}: Error finding container 2727badfe5fd7d8633b6fb610d49d0f4cc416f3fbaf3c2c6321ebeabecc5c2f7: Status 404 returned error can't find the container with id 2727badfe5fd7d8633b6fb610d49d0f4cc416f3fbaf3c2c6321ebeabecc5c2f7 Jul 22 02:21:49.333574 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:49.333477 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" event={"ID":"2702c7fa-af5a-4454-a7eb-d04b25790a61","Type":"ContainerStarted","Data":"fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2"} Jul 22 02:21:49.333574 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:49.333521 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" event={"ID":"2702c7fa-af5a-4454-a7eb-d04b25790a61","Type":"ContainerStarted","Data":"2727badfe5fd7d8633b6fb610d49d0f4cc416f3fbaf3c2c6321ebeabecc5c2f7"} Jul 22 02:21:49.354813 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:49.354767 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" podStartSLOduration=1.354751815 podStartE2EDuration="1.354751815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:49.353460148 +0000 UTC m=+2683.850901506" watchObservedRunningTime="2026-07-22 02:21:49.354751815 +0000 UTC m=+2683.852193171" Jul 22 02:21:52.193515 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:52.193476 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:21:59.009371 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:59.009281 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:59.009371 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:59.009332 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:59.021708 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:59.021680 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:21:59.384328 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:21:59.384302 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:22:00.877075 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:00.877041 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:00.976331 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:00.976294 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:02.030154 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:02.030117 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:02.131881 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:02.131844 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:05.984104 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:05.984077 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:22:05.985892 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:05.985871 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jgthx_0c717bdb-d2b6-4bf6-84fd-1b0e9d0ba082/ovn-acl-logging/0.log" Jul 22 02:22:09.102196 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:09.102157 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:22:10.046580 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:10.046540 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:10.150165 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:10.150129 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:24.489599 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:24.489567 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:22:32.401181 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:32.401139 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48"] Jul 22 02:22:32.437871 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:32.437836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:33.184893 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:33.184857 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72"] Jul 22 02:22:33.212671 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:33.212640 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:39.392925 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:39.392888 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:22:47.420622 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.420559 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" podUID="11d4b967-185b-40d8-8286-239b512acc4a" containerName="main" containerID="cri-o://d7d67f37e18faf4c720729ebd63651daabb09665a2a754a7e652cbe1310d3d66" gracePeriod=45 Jul 22 02:22:47.571345 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.571308 2570 generic.go:356] "Generic (PLEG): container finished" podID="11d4b967-185b-40d8-8286-239b512acc4a" containerID="d7d67f37e18faf4c720729ebd63651daabb09665a2a754a7e652cbe1310d3d66" exitCode=0 Jul 22 02:22:47.571523 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.571382 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" event={"ID":"11d4b967-185b-40d8-8286-239b512acc4a","Type":"ContainerDied","Data":"d7d67f37e18faf4c720729ebd63651daabb09665a2a754a7e652cbe1310d3d66"} Jul 22 02:22:47.668272 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.668246 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:22:47.715137 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715040 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-dshm\") pod \"11d4b967-185b-40d8-8286-239b512acc4a\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " Jul 22 02:22:47.715137 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715113 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-home\") pod \"11d4b967-185b-40d8-8286-239b512acc4a\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " Jul 22 02:22:47.715409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715198 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11d4b967-185b-40d8-8286-239b512acc4a-tls-certs\") pod \"11d4b967-185b-40d8-8286-239b512acc4a\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " Jul 22 02:22:47.715409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715253 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-model-cache\") pod \"11d4b967-185b-40d8-8286-239b512acc4a\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " Jul 22 02:22:47.715409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715293 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-tmp-dir\") pod \"11d4b967-185b-40d8-8286-239b512acc4a\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " Jul 22 02:22:47.715409 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715334 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f5km\" (UniqueName: \"kubernetes.io/projected/11d4b967-185b-40d8-8286-239b512acc4a-kube-api-access-6f5km\") pod \"11d4b967-185b-40d8-8286-239b512acc4a\" (UID: \"11d4b967-185b-40d8-8286-239b512acc4a\") " Jul 22 02:22:47.715696 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715521 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-model-cache" (OuterVolumeSpecName: "model-cache") pod "11d4b967-185b-40d8-8286-239b512acc4a" (UID: "11d4b967-185b-40d8-8286-239b512acc4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:47.715696 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715583 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-home" (OuterVolumeSpecName: "home") pod "11d4b967-185b-40d8-8286-239b512acc4a" (UID: "11d4b967-185b-40d8-8286-239b512acc4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:47.715696 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715655 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11d4b967-185b-40d8-8286-239b512acc4a" (UID: "11d4b967-185b-40d8-8286-239b512acc4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:47.715696 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715674 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:47.715696 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.715692 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:47.717480 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.717458 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-dshm" (OuterVolumeSpecName: "dshm") pod "11d4b967-185b-40d8-8286-239b512acc4a" (UID: "11d4b967-185b-40d8-8286-239b512acc4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:47.717584 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.717530 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d4b967-185b-40d8-8286-239b512acc4a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11d4b967-185b-40d8-8286-239b512acc4a" (UID: "11d4b967-185b-40d8-8286-239b512acc4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:22:47.717695 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.717679 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d4b967-185b-40d8-8286-239b512acc4a-kube-api-access-6f5km" (OuterVolumeSpecName: "kube-api-access-6f5km") pod "11d4b967-185b-40d8-8286-239b512acc4a" (UID: "11d4b967-185b-40d8-8286-239b512acc4a"). InnerVolumeSpecName "kube-api-access-6f5km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:47.817028 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.816992 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5km\" (UniqueName: \"kubernetes.io/projected/11d4b967-185b-40d8-8286-239b512acc4a-kube-api-access-6f5km\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:47.817028 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.817021 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:47.817028 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.817030 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11d4b967-185b-40d8-8286-239b512acc4a-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:47.817028 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:47.817038 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11d4b967-185b-40d8-8286-239b512acc4a-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:48.204099 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.204041 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" podUID="2702c7fa-af5a-4454-a7eb-d04b25790a61" containerName="main" containerID="cri-o://fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2" gracePeriod=45 Jul 22 02:22:48.430967 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.430937 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:22:48.523825 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.523740 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-model-cache\") pod \"2702c7fa-af5a-4454-a7eb-d04b25790a61\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " Jul 22 02:22:48.523825 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.523777 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-tmp-dir\") pod \"2702c7fa-af5a-4454-a7eb-d04b25790a61\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " Jul 22 02:22:48.523825 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.523815 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-dshm\") pod \"2702c7fa-af5a-4454-a7eb-d04b25790a61\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " Jul 22 02:22:48.524075 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.523837 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-home\") pod \"2702c7fa-af5a-4454-a7eb-d04b25790a61\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " Jul 22 02:22:48.524075 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.523898 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2tpm\" (UniqueName: \"kubernetes.io/projected/2702c7fa-af5a-4454-a7eb-d04b25790a61-kube-api-access-v2tpm\") pod \"2702c7fa-af5a-4454-a7eb-d04b25790a61\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " Jul 22 02:22:48.524075 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.523923 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2702c7fa-af5a-4454-a7eb-d04b25790a61-tls-certs\") pod \"2702c7fa-af5a-4454-a7eb-d04b25790a61\" (UID: \"2702c7fa-af5a-4454-a7eb-d04b25790a61\") " Jul 22 02:22:48.524253 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.524052 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-model-cache" (OuterVolumeSpecName: "model-cache") pod "2702c7fa-af5a-4454-a7eb-d04b25790a61" (UID: "2702c7fa-af5a-4454-a7eb-d04b25790a61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:48.524253 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.524127 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2702c7fa-af5a-4454-a7eb-d04b25790a61" (UID: "2702c7fa-af5a-4454-a7eb-d04b25790a61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:48.524253 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.524145 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-home" (OuterVolumeSpecName: "home") pod "2702c7fa-af5a-4454-a7eb-d04b25790a61" (UID: "2702c7fa-af5a-4454-a7eb-d04b25790a61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:48.526058 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.526028 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2702c7fa-af5a-4454-a7eb-d04b25790a61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2702c7fa-af5a-4454-a7eb-d04b25790a61" (UID: "2702c7fa-af5a-4454-a7eb-d04b25790a61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:22:48.526171 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.526066 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-dshm" (OuterVolumeSpecName: "dshm") pod "2702c7fa-af5a-4454-a7eb-d04b25790a61" (UID: "2702c7fa-af5a-4454-a7eb-d04b25790a61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:48.526171 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.526089 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2702c7fa-af5a-4454-a7eb-d04b25790a61-kube-api-access-v2tpm" (OuterVolumeSpecName: "kube-api-access-v2tpm") pod "2702c7fa-af5a-4454-a7eb-d04b25790a61" (UID: "2702c7fa-af5a-4454-a7eb-d04b25790a61"). InnerVolumeSpecName "kube-api-access-v2tpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:48.580245 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.579121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" event={"ID":"11d4b967-185b-40d8-8286-239b512acc4a","Type":"ContainerDied","Data":"9e8d6cf9e3256bbd54c7771362da7aea6989d7268c0e5c35f00c1cb0da6dda54"} Jul 22 02:22:48.580245 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.579183 2570 scope.go:117] "RemoveContainer" containerID="d7d67f37e18faf4c720729ebd63651daabb09665a2a754a7e652cbe1310d3d66" Jul 22 02:22:48.580245 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.579392 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48" Jul 22 02:22:48.581424 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.581397 2570 generic.go:356] "Generic (PLEG): container finished" podID="2702c7fa-af5a-4454-a7eb-d04b25790a61" containerID="fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2" exitCode=0 Jul 22 02:22:48.581519 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.581449 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" Jul 22 02:22:48.581519 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.581464 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" event={"ID":"2702c7fa-af5a-4454-a7eb-d04b25790a61","Type":"ContainerDied","Data":"fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2"} Jul 22 02:22:48.581519 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.581492 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72" event={"ID":"2702c7fa-af5a-4454-a7eb-d04b25790a61","Type":"ContainerDied","Data":"2727badfe5fd7d8633b6fb610d49d0f4cc416f3fbaf3c2c6321ebeabecc5c2f7"} Jul 22 02:22:48.589579 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.589561 2570 scope.go:117] "RemoveContainer" containerID="fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2" Jul 22 02:22:48.597190 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.597170 2570 scope.go:117] "RemoveContainer" containerID="fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2" Jul 22 02:22:48.597473 ip-10-0-130-174 kubenswrapper[2570]: E0722 02:22:48.597455 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2\": container with ID starting with fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2 not found: ID does not exist" containerID="fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2" Jul 22 02:22:48.597533 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.597482 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2"} err="failed to get container status \"fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2\": rpc error: code = NotFound desc = could not find container \"fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2\": container with ID starting with fac33c1312c18f041d8fdc16ea941c707a5fed41583c036654f879cf66df9fa2 not found: ID does not exist" Jul 22 02:22:48.605039 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.605015 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48"] Jul 22 02:22:48.609585 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.609560 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-eba9c8/rate-test-model-eba9c8-kserve-5896d6d96c-ldf48"] Jul 22 02:22:48.619952 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.619925 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72"] Jul 22 02:22:48.624135 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.624113 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6960c/rate-test-model-c6960c-kserve-55bdf76547-msq72"] Jul 22 02:22:48.624450 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.624434 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-dshm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:48.624501 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.624454 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-home\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:48.624501 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.624463 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tpm\" (UniqueName: \"kubernetes.io/projected/2702c7fa-af5a-4454-a7eb-d04b25790a61-kube-api-access-v2tpm\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:48.624501 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.624473 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2702c7fa-af5a-4454-a7eb-d04b25790a61-tls-certs\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:48.624501 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.624483 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-model-cache\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:48.624625 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:48.624515 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2702c7fa-af5a-4454-a7eb-d04b25790a61-tmp-dir\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:49.597373 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.597340 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d4b967-185b-40d8-8286-239b512acc4a" path="/var/lib/kubelet/pods/11d4b967-185b-40d8-8286-239b512acc4a/volumes" Jul 22 02:22:49.597742 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.597723 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2702c7fa-af5a-4454-a7eb-d04b25790a61" path="/var/lib/kubelet/pods/2702c7fa-af5a-4454-a7eb-d04b25790a61/volumes" Jul 22 02:22:49.926523 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.926489 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:49.926769 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.926757 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2702c7fa-af5a-4454-a7eb-d04b25790a61" containerName="main" Jul 22 02:22:49.926809 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.926770 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2702c7fa-af5a-4454-a7eb-d04b25790a61" containerName="main" Jul 22 02:22:49.926809 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.926788 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d4b967-185b-40d8-8286-239b512acc4a" containerName="main" Jul 22 02:22:49.926809 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.926794 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d4b967-185b-40d8-8286-239b512acc4a" containerName="main" Jul 22 02:22:49.926918 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.926838 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d4b967-185b-40d8-8286-239b512acc4a" containerName="main" Jul 22 02:22:49.926918 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.926846 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2702c7fa-af5a-4454-a7eb-d04b25790a61" containerName="main" Jul 22 02:22:49.931093 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.931076 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:49.933125 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.933105 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jp4bj\"" Jul 22 02:22:49.935400 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:49.935376 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:50.036912 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:50.036874 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttvr\" (UniqueName: \"kubernetes.io/projected/40aa607d-fc85-4458-9b63-a61e38600ae2-kube-api-access-bttvr\") pod \"test-curl-7697-139835625650192\" (UID: \"40aa607d-fc85-4458-9b63-a61e38600ae2\") " pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:50.137272 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:50.137233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttvr\" (UniqueName: \"kubernetes.io/projected/40aa607d-fc85-4458-9b63-a61e38600ae2-kube-api-access-bttvr\") pod \"test-curl-7697-139835625650192\" (UID: \"40aa607d-fc85-4458-9b63-a61e38600ae2\") " pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:50.146862 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:50.146829 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttvr\" (UniqueName: \"kubernetes.io/projected/40aa607d-fc85-4458-9b63-a61e38600ae2-kube-api-access-bttvr\") pod \"test-curl-7697-139835625650192\" (UID: \"40aa607d-fc85-4458-9b63-a61e38600ae2\") " pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:50.240960 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:50.240855 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:50.365270 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:50.365239 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:50.368775 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:22:50.368750 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40aa607d_fc85_4458_9b63_a61e38600ae2.slice/crio-e476cac6dca364b04c6920c3d750a0ab17c9157ef8555c8701f576ca7574dfed WatchSource:0}: Error finding container e476cac6dca364b04c6920c3d750a0ab17c9157ef8555c8701f576ca7574dfed: Status 404 returned error can't find the container with id e476cac6dca364b04c6920c3d750a0ab17c9157ef8555c8701f576ca7574dfed Jul 22 02:22:50.369997 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:50.369981 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:22:50.593288 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:50.593183 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625650192" event={"ID":"40aa607d-fc85-4458-9b63-a61e38600ae2","Type":"ContainerStarted","Data":"e476cac6dca364b04c6920c3d750a0ab17c9157ef8555c8701f576ca7574dfed"} Jul 22 02:22:51.598741 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:51.598707 2570 generic.go:356] "Generic (PLEG): container finished" podID="40aa607d-fc85-4458-9b63-a61e38600ae2" containerID="4cb9f7dbc336ca872f0d75d4c177daa5194fa717048ebdec501e99f6a5e4a07a" exitCode=0 Jul 22 02:22:51.599137 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:51.598763 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625650192" event={"ID":"40aa607d-fc85-4458-9b63-a61e38600ae2","Type":"ContainerDied","Data":"4cb9f7dbc336ca872f0d75d4c177daa5194fa717048ebdec501e99f6a5e4a07a"} Jul 22 02:22:52.731790 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:52.731758 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:52.752293 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:52.752269 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-139835625650192_40aa607d-fc85-4458-9b63-a61e38600ae2/test-curl-7697-139835625650192/0.log" Jul 22 02:22:52.774253 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:52.774208 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:52.781496 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:52.781471 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:52.860543 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:52.860462 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bttvr\" (UniqueName: \"kubernetes.io/projected/40aa607d-fc85-4458-9b63-a61e38600ae2-kube-api-access-bttvr\") pod \"40aa607d-fc85-4458-9b63-a61e38600ae2\" (UID: \"40aa607d-fc85-4458-9b63-a61e38600ae2\") " Jul 22 02:22:52.862657 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:52.862636 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40aa607d-fc85-4458-9b63-a61e38600ae2-kube-api-access-bttvr" (OuterVolumeSpecName: "kube-api-access-bttvr") pod "40aa607d-fc85-4458-9b63-a61e38600ae2" (UID: "40aa607d-fc85-4458-9b63-a61e38600ae2"). InnerVolumeSpecName "kube-api-access-bttvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:52.966504 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:52.962454 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bttvr\" (UniqueName: \"kubernetes.io/projected/40aa607d-fc85-4458-9b63-a61e38600ae2-kube-api-access-bttvr\") on node \"ip-10-0-130-174.ec2.internal\" DevicePath \"\"" Jul 22 02:22:53.546041 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:53.545997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:53.599466 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:53.599436 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40aa607d-fc85-4458-9b63-a61e38600ae2" path="/var/lib/kubelet/pods/40aa607d-fc85-4458-9b63-a61e38600ae2/volumes" Jul 22 02:22:53.609592 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:53.609573 2570 scope.go:117] "RemoveContainer" containerID="4cb9f7dbc336ca872f0d75d4c177daa5194fa717048ebdec501e99f6a5e4a07a" Jul 22 02:22:53.609727 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:53.609594 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:53.646596 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:22:53.646564 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:03.567276 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:03.567238 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:06.492408 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:06.492373 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:23:08.074952 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:08.074900 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:08.512250 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:08.512186 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:14.557326 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:14.557286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:14.622025 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:14.621987 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:14.880421 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:14.880390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:15.159246 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:15.159114 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:15.867180 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:15.867145 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:15.967469 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:15.967346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:25.865146 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:25.865104 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:28.696847 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:28.696755 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kwdcn"] Jul 22 02:23:30.373711 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:30.373667 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:30.867837 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:30.867803 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:36.681860 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:36.681818 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:37.276782 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:37.276743 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:37.376756 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:37.376723 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:37.615425 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:37.615397 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:37.868345 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:37.868244 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:38.381059 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:38.381027 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f54c4b9cf-tggs9_010629d1-ad00-4592-8ed7-528e9e8be5ef/authorino/0.log" Jul 22 02:23:45.078210 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:45.078179 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c578697c6-jqhsf_77612cb4-fe8d-4092-bf08-73dfb2dd58b1/copy-manifests/0.log" Jul 22 02:23:45.086610 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:45.086587 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c578697c6-jqhsf_77612cb4-fe8d-4092-bf08-73dfb2dd58b1/manager/0.log" Jul 22 02:23:47.645817 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:47.645778 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f54c4b9cf-tggs9_010629d1-ad00-4592-8ed7-528e9e8be5ef/authorino/0.log" Jul 22 02:23:48.064132 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:48.064057 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-tnwm5_52d390f5-a778-4027-8413-3162d947a38d/kuadrant-console-plugin/0.log" Jul 22 02:23:48.496285 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:48.496255 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-kwdcn_5f983e51-63a3-4406-bef9-9acb80a726c1/limitador/0.log" Jul 22 02:23:49.446563 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:49.446525 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5975cd7b94-vhl8r_65e18150-c9ce-46ff-9d14-8f5dc63c5de4/kube-auth-proxy/0.log" Jul 22 02:23:50.658191 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:50.658161 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-hn88h_6f73490b-4058-4566-ac39-ecdce85f4481/main/0.log" Jul 22 02:23:51.057452 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:51.057350 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bbq72_8e443d2c-6456-49b9-ba3f-fe5a56b936dd/main/0.log" Jul 22 02:23:57.566827 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:57.566785 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:57.664856 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:23:57.664822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" limit=1024 Jul 22 02:24:00.474873 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:00.474821 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7tb7b_21b6ba48-2cfc-4b03-a77d-0ced99379d04/global-pull-secret-syncer/0.log" Jul 22 02:24:00.626107 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:00.626047 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5nkpp_1aa63b9d-84e0-4120-a6c3-75b7c6c1b929/konnectivity-agent/0.log" Jul 22 02:24:00.724261 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:00.724230 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-174.ec2.internal_391551e060e624dcd7b1ada7a7b8652e/haproxy/0.log" Jul 22 02:24:05.311947 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:05.311913 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f54c4b9cf-tggs9_010629d1-ad00-4592-8ed7-528e9e8be5ef/authorino/0.log" Jul 22 02:24:05.426155 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:05.426129 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-tnwm5_52d390f5-a778-4027-8413-3162d947a38d/kuadrant-console-plugin/0.log" Jul 22 02:24:05.715877 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:05.715849 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-kwdcn_5f983e51-63a3-4406-bef9-9acb80a726c1/limitador/0.log" Jul 22 02:24:08.331872 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:08.331840 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c97kn_cdd794b8-ad2e-44a5-a88f-e25a2fb432a1/node-exporter/0.log" Jul 22 02:24:08.373746 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:08.373712 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c97kn_cdd794b8-ad2e-44a5-a88f-e25a2fb432a1/kube-rbac-proxy/0.log" Jul 22 02:24:08.421283 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:08.421253 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c97kn_cdd794b8-ad2e-44a5-a88f-e25a2fb432a1/init-textfile/0.log" Jul 22 02:24:10.115643 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.115606 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc"] Jul 22 02:24:10.116005 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.115877 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40aa607d-fc85-4458-9b63-a61e38600ae2" containerName="test-curl-7697-139835625650192" Jul 22 02:24:10.116005 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.115888 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="40aa607d-fc85-4458-9b63-a61e38600ae2" containerName="test-curl-7697-139835625650192" Jul 22 02:24:10.116005 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.115950 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="40aa607d-fc85-4458-9b63-a61e38600ae2" containerName="test-curl-7697-139835625650192" Jul 22 02:24:10.118832 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.118815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.121695 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.121676 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-knlbt\"/\"default-dockercfg-9xmg5\"" Jul 22 02:24:10.122666 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.122647 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-knlbt\"/\"openshift-service-ca.crt\"" Jul 22 02:24:10.122803 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.122719 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-knlbt\"/\"kube-root-ca.crt\"" Jul 22 02:24:10.131119 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.131093 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc"] Jul 22 02:24:10.156902 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.156866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-lib-modules\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.156902 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.156903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-proc\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.157089 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.156938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-sys\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.157089 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.156974 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-podres\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.157164 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.157073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447s4\" (UniqueName: \"kubernetes.io/projected/50269250-fd13-4ac4-8114-3165c3bdbe37-kube-api-access-447s4\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.260430 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.259790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-lib-modules\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.260430 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.259847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-proc\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.260430 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.259905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-sys\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.260430 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.259934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-podres\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.260430 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.260000 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447s4\" (UniqueName: \"kubernetes.io/projected/50269250-fd13-4ac4-8114-3165c3bdbe37-kube-api-access-447s4\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.261018 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.260993 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-lib-modules\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.261188 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.261172 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-proc\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.261355 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.261341 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-sys\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.261592 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.261576 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50269250-fd13-4ac4-8114-3165c3bdbe37-podres\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.271046 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.271020 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-447s4\" (UniqueName: \"kubernetes.io/projected/50269250-fd13-4ac4-8114-3165c3bdbe37-kube-api-access-447s4\") pod \"perf-node-gather-daemonset-87kkc\" (UID: \"50269250-fd13-4ac4-8114-3165c3bdbe37\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.429208 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.429113 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.567477 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.567444 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc"] Jul 22 02:24:10.571875 ip-10-0-130-174 kubenswrapper[2570]: W0722 02:24:10.571843 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod50269250_fd13_4ac4_8114_3165c3bdbe37.slice/crio-17f46cfe192844cc5b7ba2c401cca4655e4165d61af1a2275349a1fd03226ab8 WatchSource:0}: Error finding container 17f46cfe192844cc5b7ba2c401cca4655e4165d61af1a2275349a1fd03226ab8: Status 404 returned error can't find the container with id 17f46cfe192844cc5b7ba2c401cca4655e4165d61af1a2275349a1fd03226ab8 Jul 22 02:24:10.930824 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.930773 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" event={"ID":"50269250-fd13-4ac4-8114-3165c3bdbe37","Type":"ContainerStarted","Data":"cca2f0c5618ef36365989245f7aa5db0c29f3618ebb85c9df4d3d51170ab5d84"} Jul 22 02:24:10.931003 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.930848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" event={"ID":"50269250-fd13-4ac4-8114-3165c3bdbe37","Type":"ContainerStarted","Data":"17f46cfe192844cc5b7ba2c401cca4655e4165d61af1a2275349a1fd03226ab8"} Jul 22 02:24:10.931003 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.930880 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:10.950020 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:10.949975 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" podStartSLOduration=0.949960119 podStartE2EDuration="949.960119ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:24:10.949233855 +0000 UTC m=+2825.446675236" watchObservedRunningTime="2026-07-22 02:24:10.949960119 +0000 UTC m=+2825.447401475" Jul 22 02:24:13.910424 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:13.910395 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4qqk_da3008a4-466f-4a75-8a24-483ab8cf6f66/dns/0.log" Jul 22 02:24:13.933761 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:13.933731 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4qqk_da3008a4-466f-4a75-8a24-483ab8cf6f66/kube-rbac-proxy/0.log" Jul 22 02:24:14.097336 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:14.097306 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ms7nh_9868bf39-940c-4e86-9405-1920fc525b8a/dns-node-resolver/0.log" Jul 22 02:24:14.743926 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:14.743893 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvxs5_aebe006d-9a1d-4049-b3e1-37130f7339aa/node-ca/0.log" Jul 22 02:24:15.974942 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:15.974908 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5975cd7b94-vhl8r_65e18150-c9ce-46ff-9d14-8f5dc63c5de4/kube-auth-proxy/0.log" Jul 22 02:24:16.849973 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:16.849938 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mkp5b_2ed4451e-6eb6-43e2-b514-daae87017ad2/serve-healthcheck-canary/0.log" Jul 22 02:24:16.946140 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:16.946115 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-87kkc" Jul 22 02:24:17.579398 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:17.579366 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rh9g2_c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18/kube-rbac-proxy/0.log" Jul 22 02:24:17.603687 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:17.603643 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rh9g2_c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18/exporter/0.log" Jul 22 02:24:17.628897 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:17.628865 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rh9g2_c3cefbba-a9d8-4b50-a0a7-6bff2ef18f18/extractor/0.log" Jul 22 02:24:20.717821 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:20.717793 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5975cd7b94-vhl8r_65e18150-c9ce-46ff-9d14-8f5dc63c5de4/kube-auth-proxy/0.log" Jul 22 02:24:20.718856 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:20.718836 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c578697c6-jqhsf_77612cb4-fe8d-4092-bf08-73dfb2dd58b1/manager/0.log" Jul 22 02:24:20.754917 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:20.754894 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c578697c6-jqhsf_77612cb4-fe8d-4092-bf08-73dfb2dd58b1/copy-manifests/0.log" Jul 22 02:24:22.239677 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:22.239635 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5975cd7b94-vhl8r_65e18150-c9ce-46ff-9d14-8f5dc63c5de4/kube-auth-proxy/0.log" Jul 22 02:24:22.291131 ip-10-0-130-174 kubenswrapper[2570]: I0722 02:24:22.291104 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65486778df-gzszj_24409476-e5b3-40ad-9afe-7fd4d93a653d/manager/0.log"