Jul 15 22:31:22.136643 ip-10-0-130-194 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 22:31:22.136827 ip-10-0-130-194 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 22:31:22.136927 ip-10-0-130-194 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 22:31:22.137351 ip-10-0-130-194 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 22:31:32.356517 ip-10-0-130-194 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 22:31:32.356536 ip-10-0-130-194 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3b89b7626b1342d3b57232dcadae9e28 -- Jul 15 22:33:08.930513 ip-10-0-130-194 systemd[1]: Starting Kubernetes Kubelet... Jul 15 22:33:09.365892 ip-10-0-130-194 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 22:33:09.365892 ip-10-0-130-194 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 22:33:09.365892 ip-10-0-130-194 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 22:33:09.365892 ip-10-0-130-194 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 22:33:09.365892 ip-10-0-130-194 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 22:33:09.368831 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.368739 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 22:33:09.373835 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373816 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 22:33:09.373835 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373835 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373839 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373842 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373845 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373849 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373852 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373855 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373857 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373860 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373863 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373866 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373868 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373871 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373873 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373876 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373878 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373881 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373883 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373886 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373898 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 22:33:09.373907 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373901 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373904 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373907 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373910 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373913 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373917 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373921 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373924 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373927 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373930 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373932 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373935 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373937 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373940 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373943 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373945 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373948 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373950 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373953 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 22:33:09.374409 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373955 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373958 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373960 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373962 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373965 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373967 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373970 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373972 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373975 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373978 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373980 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373982 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373985 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373988 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373991 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373994 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.373997 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374000 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374002 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374004 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 22:33:09.374854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374007 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374009 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374012 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374014 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374018 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374020 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374023 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374025 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374028 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374030 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374033 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374036 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374038 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374040 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374043 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374046 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374048 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374050 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374053 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374055 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 22:33:09.375385 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374058 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374060 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374062 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374065 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374067 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374072 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374075 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374078 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374081 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374083 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374088 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374092 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374094 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374097 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374101 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374105 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374108 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374110 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374113 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 22:33:09.375854 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374861 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374870 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374873 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374876 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374878 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374881 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374884 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374887 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374889 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374893 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374895 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374898 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374900 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374903 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374905 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374908 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374911 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374913 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374916 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374919 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 22:33:09.376366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374922 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374925 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374928 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374931 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374933 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374935 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374938 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374940 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374943 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374947 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374950 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374953 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374955 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374958 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374960 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374963 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374965 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374968 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374970 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374973 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 22:33:09.376859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374975 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374978 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374980 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374983 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374986 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374988 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374992 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374996 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.374999 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375001 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375005 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375008 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375010 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375015 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375018 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375021 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375023 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375026 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375028 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 22:33:09.377388 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375031 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375033 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375036 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375038 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375040 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375043 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375045 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375048 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375050 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375052 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375055 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375058 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375060 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375063 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375065 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375068 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375070 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375073 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375075 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375078 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 22:33:09.377853 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375081 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375083 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375085 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375088 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375093 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375095 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375098 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375101 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375103 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375105 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375108 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375110 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375113 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375115 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375117 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375120 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375122 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375124 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375127 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 22:33:09.378351 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375129 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375529 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375534 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375537 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375540 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375542 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375545 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375547 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375551 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375554 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375556 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375559 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375564 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375567 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375570 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375572 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375575 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375577 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375580 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375583 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 22:33:09.378813 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375586 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375589 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375591 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375594 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375597 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375599 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375602 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375604 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375607 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375609 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375612 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375615 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375617 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375620 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375622 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375625 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375627 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375630 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375633 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 22:33:09.379310 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375636 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375638 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375641 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375643 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375646 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375650 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375652 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375655 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375657 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375660 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375663 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375665 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375667 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375670 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375674 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375677 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375679 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375682 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375684 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375687 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 22:33:09.379765 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375690 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375692 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375695 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375698 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375702 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375704 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375707 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375709 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375712 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375714 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375717 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375719 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375722 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375724 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375727 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375729 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375733 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375738 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375742 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 22:33:09.380263 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375744 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375748 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375751 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375754 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375757 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375760 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375762 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375765 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375768 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375770 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375773 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375775 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375777 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375780 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375782 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375785 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375787 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375789 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375792 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375795 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 22:33:09.380717 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375797 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.375799 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375875 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375883 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375890 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375894 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375899 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375902 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375906 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375910 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375915 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375918 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375921 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375925 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375928 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375931 2565 flags.go:64] FLAG: --cgroup-root="" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375934 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375937 2565 flags.go:64] FLAG: --client-ca-file="" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375940 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375943 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375947 2565 flags.go:64] FLAG: --cluster-domain="" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375950 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375953 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375957 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 22:33:09.381199 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375962 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375966 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375968 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375972 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375975 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375978 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375981 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375984 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375987 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375990 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375994 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375997 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.375999 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376002 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376005 2565 flags.go:64] FLAG: --enable-server="true" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376008 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376013 2565 flags.go:64] FLAG: --event-burst="100" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376016 2565 flags.go:64] FLAG: --event-qps="50" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376020 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376023 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376026 2565 flags.go:64] FLAG: --eviction-hard="" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376029 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376032 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376035 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376038 2565 flags.go:64] FLAG: --eviction-soft="" Jul 15 22:33:09.381793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376041 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376044 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376046 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376049 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376052 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376055 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376058 2565 flags.go:64] FLAG: --feature-gates="" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376063 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376066 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376069 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376072 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376075 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376078 2565 flags.go:64] FLAG: --help="false" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376081 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376084 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376087 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376089 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376093 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376096 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376099 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376102 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376104 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376107 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 22:33:09.382493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376110 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376113 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376117 2565 flags.go:64] FLAG: --kube-reserved="" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376119 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376122 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376125 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376128 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376131 2565 flags.go:64] FLAG: --lock-file="" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376133 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376136 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376139 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376144 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376147 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376149 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376152 2565 flags.go:64] FLAG: --logging-format="text" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376155 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376160 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376178 2565 flags.go:64] FLAG: --manifest-url="" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376181 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376186 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376189 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376193 2565 flags.go:64] FLAG: --max-pods="110" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376196 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376199 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376203 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 22:33:09.383108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376205 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376208 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376211 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376214 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376221 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376224 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376227 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376230 2565 flags.go:64] FLAG: --pod-cidr="" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376233 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376239 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376243 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376254 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376258 2565 flags.go:64] FLAG: --port="10250" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376262 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376265 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b0a7fb23c2790d3d" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376268 2565 flags.go:64] FLAG: --qos-reserved="" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376271 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376274 2565 flags.go:64] FLAG: --register-node="true" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376277 2565 flags.go:64] FLAG: --register-with-taints="" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376286 2565 flags.go:64] FLAG: --registry-burst="10" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376289 2565 flags.go:64] FLAG: --registry-qps="5" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376291 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376294 2565 flags.go:64] FLAG: --reserved-memory="" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376302 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376305 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 22:33:09.383739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376308 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376311 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376314 2565 flags.go:64] FLAG: --runonce="false" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376317 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376320 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376323 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376326 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376329 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376332 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376335 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376338 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376341 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376343 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376347 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376349 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376352 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376355 2565 flags.go:64] FLAG: --system-cgroups="" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376359 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376364 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376367 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376370 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376374 2565 flags.go:64] FLAG: --tls-min-version="" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376377 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376380 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376383 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 22:33:09.384375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376386 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376389 2565 flags.go:64] FLAG: --v="2" Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376393 2565 flags.go:64] FLAG: --version="false" Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376397 2565 flags.go:64] FLAG: --vmodule="" Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376400 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.376406 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376525 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376528 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376531 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376533 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376536 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376538 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376541 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376544 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376546 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376549 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376551 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376554 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376556 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376558 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376561 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 22:33:09.385254 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376564 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376566 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376569 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376572 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376575 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376578 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376580 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376583 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376586 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376588 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376591 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376593 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376596 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376598 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376601 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376604 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376608 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376610 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376613 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376616 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 22:33:09.386020 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376618 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376621 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376624 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376626 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376629 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376631 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376634 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376636 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376639 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376642 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376644 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376647 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376650 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376652 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376655 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376659 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376661 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376664 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376666 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 22:33:09.386557 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376668 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376671 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376673 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376676 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376678 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376680 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376683 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376685 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376688 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376692 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376694 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376697 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376700 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376702 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376704 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376707 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376711 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376714 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376716 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 22:33:09.387071 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376720 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376724 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376727 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376730 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376732 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376735 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376737 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376740 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376742 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376746 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376748 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376751 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376754 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376756 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376759 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376761 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376764 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376767 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376769 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376772 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 22:33:09.387593 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376774 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376777 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376783 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376786 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376789 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.376791 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.377660 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.386227 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.386244 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.387152 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.387178 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387238 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387244 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387247 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 22:33:09.388084 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387250 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387253 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387255 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387258 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387261 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387263 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387266 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387268 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387272 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387275 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387277 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387281 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387283 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387286 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387288 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387291 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387293 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387296 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387298 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387301 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 22:33:09.388454 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387304 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387307 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387309 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387313 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387318 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387321 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387324 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387327 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387330 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387333 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387336 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387338 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387341 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387344 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387346 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387349 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387351 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387354 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387356 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 22:33:09.388934 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387359 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387362 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387364 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387367 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387371 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387374 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387376 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387378 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387381 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387383 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387386 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387389 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387391 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387394 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387398 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387402 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387405 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387408 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387411 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 22:33:09.389401 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387413 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387416 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387418 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387421 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387423 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387426 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387428 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387431 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387433 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387435 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387438 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387440 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387442 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387445 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387448 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387451 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387453 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387456 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387460 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387463 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 22:33:09.389852 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387465 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387468 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387470 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387473 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387475 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387478 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387480 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387483 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387485 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387488 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387491 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387493 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387496 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387498 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387500 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387503 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387505 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 22:33:09.390336 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387508 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.387513 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387653 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387659 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387664 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387667 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387670 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387673 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387676 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387679 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387682 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387685 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387688 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387691 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387694 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 22:33:09.390739 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387697 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387700 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387702 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387705 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387708 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387712 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387715 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387718 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387721 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387724 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387726 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387729 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387732 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387734 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387737 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387739 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387742 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387744 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387747 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 22:33:09.391105 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387750 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387752 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387754 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387757 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387759 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387762 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387764 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387767 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387770 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387773 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387775 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387778 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387780 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387783 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387786 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387788 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387791 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387793 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387795 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387799 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 22:33:09.391698 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387801 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387804 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387806 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387809 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387811 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387813 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387816 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387819 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387821 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387823 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387826 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387828 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387831 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387834 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387836 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387838 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387841 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387843 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387845 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387848 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 22:33:09.392178 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387850 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387853 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387855 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387858 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387860 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387862 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387866 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387868 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387871 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387873 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387875 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387878 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387881 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387883 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387885 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387887 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387890 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387893 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387897 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 22:33:09.392738 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387899 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387902 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387905 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387908 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387910 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387913 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387916 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:09.387918 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.387922 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.388043 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 22:33:09.393207 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.392234 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 22:33:09.394123 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.394110 2565 server.go:1019] "Starting client certificate rotation" Jul 15 22:33:09.394235 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.394219 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 22:33:09.394277 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.394252 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 22:33:09.397222 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.397208 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 22:33:09.401151 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.401132 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 22:33:09.415299 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.415275 2565 log.go:25] "Validated CRI v1 runtime API" Jul 15 22:33:09.417776 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.417759 2565 log.go:25] "Validated CRI v1 image API" Jul 15 22:33:09.418586 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.418566 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 22:33:09.422126 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.422091 2565 fs.go:135] Filesystem UUIDs: map[3983cf59-d149-42f2-9081-72ef5fc8aed3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 907c5c01-ecf4-445a-82a5-f9868cffc333:/dev/nvme0n1p3] Jul 15 22:33:09.422227 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.422124 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 22:33:09.425368 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.425339 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 22:33:09.429020 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.428888 2565 manager.go:216] Machine: {Timestamp:2026-07-15 22:33:09.427433053 +0000 UTC m=+0.367635594 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110443 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2488fc56fc6a1881add738ca123c41 SystemUUID:ec2488fc-56fc-6a18-81ad-d738ca123c41 BootID:3b89b762-6b13-42d3-b572-32dcadae9e28 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:97:97:4f:44:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:97:97:4f:44:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:19:bd:c5:4c:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 22:33:09.429020 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.429010 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 22:33:09.429219 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.429146 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 22:33:09.432991 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.432955 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 22:33:09.433184 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.432993 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 22:33:09.433915 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.433903 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 22:33:09.433951 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.433918 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 22:33:09.433951 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.433926 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 22:33:09.433951 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.433943 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 22:33:09.435869 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.435858 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 22:33:09.440676 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.440663 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 15 22:33:09.440804 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.440794 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 22:33:09.440842 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.440825 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 15 22:33:09.440842 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.440840 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 22:33:09.440903 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.440855 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 22:33:09.440903 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.440865 2565 kubelet.go:398] "Adding apiserver pod source" Jul 15 22:33:09.440903 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.440873 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 22:33:09.442202 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.442157 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 22:33:09.442306 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.442215 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 22:33:09.444988 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.444965 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 22:33:09.447446 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.447431 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 22:33:09.447518 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.447462 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 22:33:09.447681 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.447670 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 22:33:09.447716 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.447686 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 22:33:09.447716 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.447692 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 22:33:09.447716 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.447698 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 22:33:09.447716 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.447707 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 22:33:09.447716 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.447714 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 22:33:09.448236 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.448226 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 22:33:09.448285 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.448236 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 22:33:09.448285 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.448244 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 22:33:09.448285 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.448258 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 22:33:09.448285 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.448268 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 22:33:09.448285 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.448276 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 22:33:09.449207 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.449193 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 22:33:09.449207 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.449207 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 22:33:09.451329 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.451306 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 22:33:09.451396 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.451340 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 22:33:09.451433 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.451406 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 22:33:09.452893 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.452880 2565 server.go:1268] "Started kubelet" Jul 15 22:33:09.452974 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.452948 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 22:33:09.453021 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.452964 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 22:33:09.453065 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.453034 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 22:33:09.453955 ip-10-0-130-194 systemd[1]: Started Kubernetes Kubelet. Jul 15 22:33:09.454625 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.454611 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 22:33:09.455644 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.455631 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 15 22:33:09.459030 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.459009 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 22:33:09.461330 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.460131 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-194.ec2.internal.18c2975401a61fd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-194.ec2.internal,UID:ip-10-0-130-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-194.ec2.internal,},FirstTimestamp:2026-07-15 22:33:09.452849109 +0000 UTC m=+0.393051650,LastTimestamp:2026-07-15 22:33:09.452849109 +0000 UTC m=+0.393051650,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-194.ec2.internal,}" Jul 15 22:33:09.462118 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.462100 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 22:33:09.462118 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.462108 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 22:33:09.462284 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.462216 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:09.462928 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.462910 2565 factory.go:55] Registering systemd factory Jul 15 22:33:09.463025 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.462931 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 22:33:09.463025 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.462934 2565 factory.go:223] Registration of the systemd container factory successfully Jul 15 22:33:09.463025 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.462931 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 22:33:09.463025 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.462992 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 22:33:09.463237 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.463221 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 15 22:33:09.463237 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.463232 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 15 22:33:09.463237 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.463232 2565 factory.go:153] Registering CRI-O factory Jul 15 22:33:09.463330 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.463249 2565 factory.go:223] Registration of the crio container factory successfully Jul 15 22:33:09.463330 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.463324 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 22:33:09.463387 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.463351 2565 factory.go:103] Registering Raw factory Jul 15 22:33:09.463387 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.463367 2565 manager.go:1196] Started watching for new ooms in manager Jul 15 22:33:09.463824 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.463807 2565 manager.go:349] Starting recovery of all containers Jul 15 22:33:09.471766 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.467988 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 22:33:09.471766 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.468036 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 22:33:09.471766 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.470540 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sd2xz" Jul 15 22:33:09.476273 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.476238 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sd2xz" Jul 15 22:33:09.476825 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.476562 2565 manager.go:354] Recovery completed Jul 15 22:33:09.480532 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.480519 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 22:33:09.482889 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.482874 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory" Jul 15 22:33:09.482951 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.482906 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 22:33:09.482951 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.482918 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID" Jul 15 22:33:09.483472 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.483457 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 22:33:09.483472 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.483469 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 22:33:09.483604 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.483484 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 15 22:33:09.483604 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.483503 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 22:33:09.484298 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.484284 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory" Jul 15 22:33:09.484342 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.484312 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 22:33:09.484342 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.484322 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID" Jul 15 22:33:09.484990 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.484913 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-194.ec2.internal.18c297540370811e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-194.ec2.internal,UID:ip-10-0-130-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-194.ec2.internal,},FirstTimestamp:2026-07-15 22:33:09.482889502 +0000 UTC m=+0.423092043,LastTimestamp:2026-07-15 22:33:09.482889502 +0000 UTC m=+0.423092043,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-194.ec2.internal,}" Jul 15 22:33:09.485532 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.485519 2565 policy_none.go:49] "None policy: Start" Jul 15 22:33:09.485568 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.485536 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 22:33:09.485568 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.485546 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 22:33:09.487568 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.487556 2565 policy_none.go:47] "Start" Jul 15 22:33:09.517782 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.517762 2565 manager.go:337] "Starting Device Plugin manager" Jul 15 22:33:09.517886 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.517798 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 22:33:09.517886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.517808 2565 server.go:87] "Starting device plugin registration server" Jul 15 22:33:09.518127 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.518111 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 22:33:09.518255 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.518126 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 22:33:09.518332 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.518312 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 22:33:09.518414 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.518403 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 22:33:09.518414 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.518414 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 22:33:09.518862 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.518834 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 22:33:09.518955 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.518901 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:09.576941 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.576907 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 22:33:09.579149 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.578434 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 22:33:09.579149 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.578474 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 22:33:09.579149 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.578502 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 22:33:09.579149 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.578541 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 22:33:09.581256 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.581236 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 22:33:09.618807 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.618721 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 22:33:09.619939 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.619912 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory" Jul 15 22:33:09.620055 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.619945 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 22:33:09.620055 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.619961 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID" Jul 15 22:33:09.620055 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.619992 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.629806 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.629779 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.629928 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.629810 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-194.ec2.internal\": node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:09.657984 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.657952 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:09.678817 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.678783 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"] Jul 15 22:33:09.678960 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.678854 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 22:33:09.680390 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.680372 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory" Jul 15 22:33:09.680514 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.680401 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 22:33:09.680514 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.680411 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID" Jul 15 22:33:09.682767 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.682753 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 22:33:09.682878 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.682862 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.682919 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.682910 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 22:33:09.683565 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.683548 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory" Jul 15 22:33:09.683670 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.683587 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory" Jul 15 22:33:09.683670 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.683612 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 22:33:09.683670 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.683628 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID" Jul 15 22:33:09.683670 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.683593 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 22:33:09.683670 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.683669 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID" Jul 15 22:33:09.685871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.685846 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.685871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.685874 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 22:33:09.686559 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.686544 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientMemory" Jul 15 22:33:09.686628 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.686571 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 22:33:09.686628 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.686581 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeHasSufficientPID" Jul 15 22:33:09.716129 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.716105 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-194.ec2.internal\" not found" node="ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.720747 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.720726 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-194.ec2.internal\" not found" node="ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.759064 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.759033 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:09.764352 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.764329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/010e8802e63dc57e897d2d31228c5508-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"010e8802e63dc57e897d2d31228c5508\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.764430 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.764360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/010e8802e63dc57e897d2d31228c5508-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"010e8802e63dc57e897d2d31228c5508\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.764430 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.764388 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/433ef3a1b26ec88b78ef4885409e47ef-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"433ef3a1b26ec88b78ef4885409e47ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.859319 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.859276 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:09.864611 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.864591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/010e8802e63dc57e897d2d31228c5508-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"010e8802e63dc57e897d2d31228c5508\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.864651 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.864620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/010e8802e63dc57e897d2d31228c5508-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"010e8802e63dc57e897d2d31228c5508\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.864651 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.864642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/433ef3a1b26ec88b78ef4885409e47ef-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"433ef3a1b26ec88b78ef4885409e47ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.864743 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.864668 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/433ef3a1b26ec88b78ef4885409e47ef-config\") pod \"kube-apiserver-proxy-ip-10-0-130-194.ec2.internal\" (UID: \"433ef3a1b26ec88b78ef4885409e47ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.864743 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.864693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/010e8802e63dc57e897d2d31228c5508-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"010e8802e63dc57e897d2d31228c5508\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.864824 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:09.864709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/010e8802e63dc57e897d2d31228c5508-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal\" (UID: \"010e8802e63dc57e897d2d31228c5508\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:09.960137 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:09.960058 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:10.018564 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.018535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:10.023742 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.023537 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" Jul 15 22:33:10.060241 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:10.060211 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:10.160765 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:10.160712 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:10.261349 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:10.261259 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:10.362096 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:10.362059 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-194.ec2.internal\" not found" Jul 15 22:33:10.394579 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.394542 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 22:33:10.395289 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.394716 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 22:33:10.442800 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.442767 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 22:33:10.453565 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.453538 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 22:33:10.462247 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.462217 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 22:33:10.462247 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.462238 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" Jul 15 22:33:10.481501 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.481441 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 22:28:10 +0000 UTC" deadline="2028-03-02 03:55:59.557319392 +0000 UTC" Jul 15 22:33:10.481501 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.481493 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14285h22m49.075829927s" Jul 15 22:33:10.481711 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.481578 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 22:33:10.483334 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.483319 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 22:33:10.484927 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.484911 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" Jul 15 22:33:10.492299 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.492284 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 22:33:10.502711 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.502688 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xdfrg" Jul 15 22:33:10.510012 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.509982 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xdfrg" Jul 15 22:33:10.519928 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.519855 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 22:33:10.690911 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:10.690868 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010e8802e63dc57e897d2d31228c5508.slice/crio-eb497d97bfd1f0458ba4dd9e126ec927622f467238a0631b93023c4a86137c8b WatchSource:0}: Error finding container eb497d97bfd1f0458ba4dd9e126ec927622f467238a0631b93023c4a86137c8b: Status 404 returned error can't find the container with id eb497d97bfd1f0458ba4dd9e126ec927622f467238a0631b93023c4a86137c8b Jul 15 22:33:10.698660 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:10.698642 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 22:33:10.704308 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:10.704284 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ef3a1b26ec88b78ef4885409e47ef.slice/crio-864fd6643f10d1326f21c3d8c2e882431b0dde4378c5a0a1944dacc2226cb6b9 WatchSource:0}: Error finding container 864fd6643f10d1326f21c3d8c2e882431b0dde4378c5a0a1944dacc2226cb6b9: Status 404 returned error can't find the container with id 864fd6643f10d1326f21c3d8c2e882431b0dde4378c5a0a1944dacc2226cb6b9 Jul 15 22:33:11.443354 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.443318 2565 apiserver.go:52] "Watching apiserver" Jul 15 22:33:11.455014 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.454977 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 22:33:11.455527 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.455496 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-7xskg","openshift-image-registry/node-ca-jfgkz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal","openshift-multus/multus-additional-cni-plugins-6bx82","openshift-multus/network-metrics-daemon-jhmzm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj","openshift-dns/node-resolver-hdrnr","openshift-multus/multus-pk7xr","openshift-network-diagnostics/network-check-target-v26ns","openshift-network-operator/iptables-alerter-6vt2l","openshift-ovn-kubernetes/ovnkube-node-qj5gc","kube-system/global-pull-secret-syncer-gp62w","kube-system/konnectivity-agent-5kgf2","kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal"] Jul 15 22:33:11.458303 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.458275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.460684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.460657 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:11.461470 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.461444 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 22:33:11.461591 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.461574 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 22:33:11.461659 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.461584 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5hwx8\"" Jul 15 22:33:11.463069 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.463045 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.464363 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.464328 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 22:33:11.464475 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.464428 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p4dcz\"" Jul 15 22:33:11.464542 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.464474 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 22:33:11.467222 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.466298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.467222 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.466501 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 22:33:11.467222 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.466790 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 22:33:11.467222 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.466881 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 22:33:11.467222 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.466891 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 22:33:11.467222 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.467065 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f2xcr\"" Jul 15 22:33:11.471412 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.471391 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 22:33:11.471527 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.471424 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 22:33:11.471527 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.471455 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x2cfq\"" Jul 15 22:33:11.471753 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.471730 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:11.472809 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.471857 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:11.473047 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473026 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-socket-dir-parent\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473125 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473051 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.473125 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-cni-bin\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473125 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbtx\" (UniqueName: \"kubernetes.io/projected/27f508ca-8bf5-4bfb-9e27-4f543593132c-kube-api-access-bdbtx\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473308 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473145 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.473308 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-hostroot\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473308 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.473459 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbpp\" (UniqueName: \"kubernetes.io/projected/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-kube-api-access-tgbpp\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.473459 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27f508ca-8bf5-4bfb-9e27-4f543593132c-cni-binary-copy\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473459 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473422 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-etc-kubernetes\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.473621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bd15363-1a4e-4956-b662-f8c68f813bd9-konnectivity-ca\") pod \"konnectivity-agent-5kgf2\" (UID: \"1bd15363-1a4e-4956-b662-f8c68f813bd9\") " pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:11.473621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-cnibin\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-k8s-cni-cncf-io\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-netns\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473626 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-multus-certs\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cnibin\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.473871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-os-release\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.473871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-system-cni-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-os-release\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.473871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85st2\" (UniqueName: \"kubernetes.io/projected/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-kube-api-access-85st2\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.473871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bd15363-1a4e-4956-b662-f8c68f813bd9-agent-certs\") pod \"konnectivity-agent-5kgf2\" (UID: \"1bd15363-1a4e-4956-b662-f8c68f813bd9\") " pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:11.473871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-cni-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.474258 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473893 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-conf-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.474258 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-daemon-config\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.474258 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.473970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-cni-multus\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.474258 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.474037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-system-cni-dir\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.474258 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.474081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.474523 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.474454 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-tmp-dir\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.474523 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.474483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-kubelet\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.474523 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.474503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-hosts-file\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.475744 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.475722 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 22:33:11.475958 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.475939 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pqf7d\"" Jul 15 22:33:11.475958 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.475946 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 22:33:11.476241 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.476209 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:11.476326 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.476245 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 22:33:11.476326 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.476291 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:11.478592 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.478570 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.481113 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.481091 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 22:33:11.481240 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.481170 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.481681 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.481391 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 22:33:11.481681 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.481536 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dm6xr\"" Jul 15 22:33:11.483749 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.483690 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.483948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.483925 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 22:33:11.484458 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.484202 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 22:33:11.484458 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.484233 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7jv2n\"" Jul 15 22:33:11.484458 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.484202 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 22:33:11.486544 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.486244 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 22:33:11.486544 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.486300 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sm94g\"" Jul 15 22:33:11.486544 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.486403 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 22:33:11.486730 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.486701 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 22:33:11.486801 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.486783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.489195 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.489173 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 22:33:11.490038 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.489507 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 22:33:11.490038 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.489587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k8fgx\"" Jul 15 22:33:11.490038 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.489607 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.490038 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.489650 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 22:33:11.490038 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.489675 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:11.490038 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.489736 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 22:33:11.490765 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.490741 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 22:33:11.490878 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.490821 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 22:33:11.513531 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.513496 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 22:28:11 +0000 UTC" deadline="2028-01-04 06:28:46.038323028 +0000 UTC" Jul 15 22:33:11.513531 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.513530 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12895h55m34.524798277s" Jul 15 22:33:11.564319 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.564288 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 22:33:11.575719 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-os-release\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.575902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85st2\" (UniqueName: \"kubernetes.io/projected/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-kube-api-access-85st2\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.575902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-systemd-units\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.575902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysctl-d\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.575902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-tuned\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.575902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysctl-conf\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.575902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-run\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-system-cni-dir\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-tmp-dir\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.575989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-lib-modules\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-tmp\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwsf\" (UniqueName: \"kubernetes.io/projected/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-kube-api-access-znwsf\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-kubelet\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576088 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-host-slash\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovn-node-metrics-cert\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovnkube-script-lib\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-socket-dir-parent\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-cni-bin\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.576253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576237 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576265 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-run-netns\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-etc-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-kubernetes\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576397 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-sys\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576422 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27f508ca-8bf5-4bfb-9e27-4f543593132c-cni-binary-copy\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576475 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-etc-kubernetes\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576492 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-os-release\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a63fe02-5823-4b44-b017-059e1e8ab33d-kubelet-config\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bd15363-1a4e-4956-b662-f8c68f813bd9-konnectivity-ca\") pod \"konnectivity-agent-5kgf2\" (UID: \"1bd15363-1a4e-4956-b662-f8c68f813bd9\") " pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-cnibin\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-k8s-cni-cncf-io\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-os-release\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.576948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d432250-ab22-419b-b783-26186a2bcfbf-serviceca\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-system-cni-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-system-cni-dir\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cvn\" (UniqueName: \"kubernetes.io/projected/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-kube-api-access-p4cvn\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bd15363-1a4e-4956-b662-f8c68f813bd9-agent-certs\") pod \"konnectivity-agent-5kgf2\" (UID: \"1bd15363-1a4e-4956-b662-f8c68f813bd9\") " pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-cni-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-conf-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-daemon-config\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-var-lib-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.576974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-host\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cppb\" (UniqueName: \"kubernetes.io/projected/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-kube-api-access-9cppb\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-cni-multus\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-tmp-dir\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-iptables-alerter-script\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-slash\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.578072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovnkube-config\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-env-overrides\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-hosts-file\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-kubelet\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577220 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-device-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbtx\" (UniqueName: \"kubernetes.io/projected/27f508ca-8bf5-4bfb-9e27-4f543593132c-kube-api-access-bdbtx\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-conf-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-hosts-file\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.577999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-daemon-config\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578074 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-kubelet\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578119 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbkjb\" (UniqueName: \"kubernetes.io/projected/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-kube-api-access-kbkjb\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578120 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-os-release\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-node-log\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-modprobe-d\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.578953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a63fe02-5823-4b44-b017-059e1e8ab33d-dbus\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-hostroot\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578265 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbpp\" (UniqueName: \"kubernetes.io/projected/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-kube-api-access-tgbpp\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578300 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-sys-fs\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj5mx\" (UniqueName: \"kubernetes.io/projected/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-kube-api-access-pj5mx\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-var-lib-kubelet\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d432250-ab22-419b-b783-26186a2bcfbf-host\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-systemd\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-cni-bin\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578425 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-cni-netd\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-socket-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578470 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysconfig\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-systemd\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.579848 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-netns\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-multus-certs\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cnibin\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-ovn\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-log-socket\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27f508ca-8bf5-4bfb-9e27-4f543593132c-cni-binary-copy\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-registration-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbpl\" (UniqueName: \"kubernetes.io/projected/5d432250-ab22-419b-b783-26186a2bcfbf-kube-api-access-wfbpl\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-system-cni-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.578933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-etc-kubernetes\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.579231 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.579609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-cni-dir\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.579636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-netns\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.579661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-hostroot\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.579715 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-k8s-cni-cncf-io\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.579775 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-cnibin\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.579812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-run-multus-certs\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.580092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cnibin\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.580758 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.580137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-cni-multus\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.581787 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.580165 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bd15363-1a4e-4956-b662-f8c68f813bd9-konnectivity-ca\") pod \"konnectivity-agent-5kgf2\" (UID: \"1bd15363-1a4e-4956-b662-f8c68f813bd9\") " pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:11.581787 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.580187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-multus-socket-dir-parent\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.581787 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.580227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cni-binary-copy\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.581787 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.580239 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27f508ca-8bf5-4bfb-9e27-4f543593132c-host-var-lib-cni-bin\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.581981 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.581884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.584364 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.584338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bd15363-1a4e-4956-b662-f8c68f813bd9-agent-certs\") pod \"konnectivity-agent-5kgf2\" (UID: \"1bd15363-1a4e-4956-b662-f8c68f813bd9\") " pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:11.584638 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.584592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" event={"ID":"433ef3a1b26ec88b78ef4885409e47ef","Type":"ContainerStarted","Data":"864fd6643f10d1326f21c3d8c2e882431b0dde4378c5a0a1944dacc2226cb6b9"} Jul 15 22:33:11.587471 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.587408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"010e8802e63dc57e897d2d31228c5508","Type":"ContainerStarted","Data":"eb497d97bfd1f0458ba4dd9e126ec927622f467238a0631b93023c4a86137c8b"} Jul 15 22:33:11.588012 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.587951 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 22:33:11.589046 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.589017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbtx\" (UniqueName: \"kubernetes.io/projected/27f508ca-8bf5-4bfb-9e27-4f543593132c-kube-api-access-bdbtx\") pod \"multus-pk7xr\" (UID: \"27f508ca-8bf5-4bfb-9e27-4f543593132c\") " pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.589305 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.589274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85st2\" (UniqueName: \"kubernetes.io/projected/1cf1c5d5-ba93-4ef4-b043-8233b97b14d3-kube-api-access-85st2\") pod \"node-resolver-hdrnr\" (UID: \"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3\") " pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.590435 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.590414 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbpp\" (UniqueName: \"kubernetes.io/projected/a449bf70-dfdc-4c4a-9248-ae820d7dbe71-kube-api-access-tgbpp\") pod \"multus-additional-cni-plugins-6bx82\" (UID: \"a449bf70-dfdc-4c4a-9248-ae820d7dbe71\") " pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.679673 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbkjb\" (UniqueName: \"kubernetes.io/projected/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-kube-api-access-kbkjb\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.679673 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-node-log\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-modprobe-d\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a63fe02-5823-4b44-b017-059e1e8ab33d-dbus\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679803 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-sys-fs\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj5mx\" (UniqueName: \"kubernetes.io/projected/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-kube-api-access-pj5mx\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-var-lib-kubelet\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.679936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d432250-ab22-419b-b783-26186a2bcfbf-host\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-systemd\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.679983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-cni-bin\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-cni-netd\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-socket-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysconfig\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-systemd\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-ovn\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-log-socket\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-registration-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680191 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbpl\" (UniqueName: \"kubernetes.io/projected/5d432250-ab22-419b-b783-26186a2bcfbf-kube-api-access-wfbpl\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-systemd-units\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680265 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysctl-d\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-tuned\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysctl-conf\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-run\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:11.680442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-lib-modules\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-tmp\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwsf\" (UniqueName: \"kubernetes.io/projected/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-kube-api-access-znwsf\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-host-slash\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680520 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovn-node-metrics-cert\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovnkube-script-lib\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-run-netns\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-etc-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-kubernetes\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-sys\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680732 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a63fe02-5823-4b44-b017-059e1e8ab33d-kubelet-config\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d432250-ab22-419b-b783-26186a2bcfbf-serviceca\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cvn\" (UniqueName: \"kubernetes.io/projected/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-kube-api-access-p4cvn\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.681356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-var-lib-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-host\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cppb\" (UniqueName: \"kubernetes.io/projected/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-kube-api-access-9cppb\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-iptables-alerter-script\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.680996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-slash\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovnkube-config\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-env-overrides\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-kubelet\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-device-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-device-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681532 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-node-log\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-lib-modules\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-modprobe-d\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681751 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a63fe02-5823-4b44-b017-059e1e8ab33d-dbus\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.682196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.681893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-host-slash\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.682947 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.682410 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-socket-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.682947 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.682513 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:11.682947 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.682603 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs podName:7b9b6737-f701-4ecb-86f1-0c25f73efdf6 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:12.18257027 +0000 UTC m=+3.122772822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs") pod "network-metrics-daemon-jhmzm" (UID: "7b9b6737-f701-4ecb-86f1-0c25f73efdf6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:11.682947 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.682817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.682947 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.682904 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.682947 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.682923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-systemd\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.683347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.682952 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysconfig\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.683347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.682983 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-ovn\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.683347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-sys-fs\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.683347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683034 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-log-socket\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.683347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-registration-dir\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.683621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683466 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysctl-conf\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.683621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683556 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-run\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-cni-bin\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d432250-ab22-419b-b783-26186a2bcfbf-host\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683847 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-systemd\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.683912 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-systemd-units\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d432250-ab22-419b-b783-26186a2bcfbf-serviceca\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684166 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-run-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-var-lib-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-host\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovnkube-script-lib\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684678 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684727 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-run-netns\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-etc-openvswitch\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684858 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-kubernetes\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.684899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-sys\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.684960 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:11.686847 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.685024 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret podName:2a63fe02-5823-4b44-b017-059e1e8ab33d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:12.185010481 +0000 UTC m=+3.125213012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret") pod "global-pull-secret-syncer-gp62w" (UID: "2a63fe02-5823-4b44-b017-059e1e8ab33d") : object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.685060 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a63fe02-5823-4b44-b017-059e1e8ab33d-kubelet-config\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.685323 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-cni-netd\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.685451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-sysctl-d\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.685902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovnkube-config\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.685968 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-slash\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.686132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-iptables-alerter-script\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.686209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-host-kubelet\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.686279 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-env-overrides\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.686407 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-var-lib-kubelet\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.686495 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-etc-tuned\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.687886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.686801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-ovn-node-metrics-cert\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.689389 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.688752 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 22:33:11.689389 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.688781 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 22:33:11.689389 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.688796 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gmbkq for pod openshift-network-diagnostics/network-check-target-v26ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:11.689389 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:11.688900 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq podName:a3257827-435d-40d8-ad19-955d7041302e nodeName:}" failed. No retries permitted until 2026-07-15 22:33:12.188881342 +0000 UTC m=+3.129083871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gmbkq" (UniqueName: "kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq") pod "network-check-target-v26ns" (UID: "a3257827-435d-40d8-ad19-955d7041302e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:11.689734 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.689713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-tmp\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.692265 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.692198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbkjb\" (UniqueName: \"kubernetes.io/projected/2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9-kube-api-access-kbkjb\") pod \"iptables-alerter-6vt2l\" (UID: \"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9\") " pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.692508 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.692439 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwsf\" (UniqueName: \"kubernetes.io/projected/1daa7794-3d7e-43b0-b053-6ff4d17f71ec-kube-api-access-znwsf\") pod \"tuned-7xskg\" (UID: \"1daa7794-3d7e-43b0-b053-6ff4d17f71ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.696229 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.696142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj5mx\" (UniqueName: \"kubernetes.io/projected/b4abc8f6-aea5-43f8-b461-f5f7bf9528d8-kube-api-access-pj5mx\") pod \"aws-ebs-csi-driver-node-6rkjj\" (UID: \"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.696414 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.696393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbpl\" (UniqueName: \"kubernetes.io/projected/5d432250-ab22-419b-b783-26186a2bcfbf-kube-api-access-wfbpl\") pod \"node-ca-jfgkz\" (UID: \"5d432250-ab22-419b-b783-26186a2bcfbf\") " pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.697128 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.697104 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cvn\" (UniqueName: \"kubernetes.io/projected/7c4945d4-04f4-480b-9abb-5db6c5a8a87c-kube-api-access-p4cvn\") pod \"ovnkube-node-qj5gc\" (UID: \"7c4945d4-04f4-480b-9abb-5db6c5a8a87c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.698729 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.698707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cppb\" (UniqueName: \"kubernetes.io/projected/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-kube-api-access-9cppb\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:11.772800 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.772760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hdrnr" Jul 15 22:33:11.780941 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.780915 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:11.788717 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.788693 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pk7xr" Jul 15 22:33:11.794830 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.794805 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6bx82" Jul 15 22:33:11.804597 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.804572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" Jul 15 22:33:11.811027 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.811002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xskg" Jul 15 22:33:11.817129 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.817106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jfgkz" Jul 15 22:33:11.824511 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.824340 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6vt2l" Jul 15 22:33:11.838393 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:11.838362 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:11.987023 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:11.986977 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4abc8f6_aea5_43f8_b461_f5f7bf9528d8.slice/crio-5b294839fbf4cadc294474a54fa528f03d039636eaf3028f5f963fa6f0e88680 WatchSource:0}: Error finding container 5b294839fbf4cadc294474a54fa528f03d039636eaf3028f5f963fa6f0e88680: Status 404 returned error can't find the container with id 5b294839fbf4cadc294474a54fa528f03d039636eaf3028f5f963fa6f0e88680 Jul 15 22:33:11.990565 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:11.990534 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d432250_ab22_419b_b783_26186a2bcfbf.slice/crio-465e58b139a44f48732cad621e91a68c961e9cc9d34c099c33b0846551b2025c WatchSource:0}: Error finding container 465e58b139a44f48732cad621e91a68c961e9cc9d34c099c33b0846551b2025c: Status 404 returned error can't find the container with id 465e58b139a44f48732cad621e91a68c961e9cc9d34c099c33b0846551b2025c Jul 15 22:33:12.015821 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:12.015786 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda449bf70_dfdc_4c4a_9248_ae820d7dbe71.slice/crio-aecd6a2378bebfd4d802af7ff552188ec272f19bdba0c50be656ba337a116b58 WatchSource:0}: Error finding container aecd6a2378bebfd4d802af7ff552188ec272f19bdba0c50be656ba337a116b58: Status 404 returned error can't find the container with id aecd6a2378bebfd4d802af7ff552188ec272f19bdba0c50be656ba337a116b58 Jul 15 22:33:12.018324 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:12.018295 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf1c5d5_ba93_4ef4_b043_8233b97b14d3.slice/crio-5c292c74b1029c214501c40ab7fe68e73fade8267720b44180352074349fb0e8 WatchSource:0}: Error finding container 5c292c74b1029c214501c40ab7fe68e73fade8267720b44180352074349fb0e8: Status 404 returned error can't find the container with id 5c292c74b1029c214501c40ab7fe68e73fade8267720b44180352074349fb0e8 Jul 15 22:33:12.020802 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:12.020560 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4945d4_04f4_480b_9abb_5db6c5a8a87c.slice/crio-7dab05ad207ab60885980912ed21ebe0a640c1affd8ece0ed5a4ab1bd9626bae WatchSource:0}: Error finding container 7dab05ad207ab60885980912ed21ebe0a640c1affd8ece0ed5a4ab1bd9626bae: Status 404 returned error can't find the container with id 7dab05ad207ab60885980912ed21ebe0a640c1affd8ece0ed5a4ab1bd9626bae Jul 15 22:33:12.021372 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:12.021265 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daa7794_3d7e_43b0_b053_6ff4d17f71ec.slice/crio-5431a571e3798281020255403aaa0513260a85663479a6d1928fd7d7f2688fd0 WatchSource:0}: Error finding container 5431a571e3798281020255403aaa0513260a85663479a6d1928fd7d7f2688fd0: Status 404 returned error can't find the container with id 5431a571e3798281020255403aaa0513260a85663479a6d1928fd7d7f2688fd0 Jul 15 22:33:12.021767 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:12.021744 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f508ca_8bf5_4bfb_9e27_4f543593132c.slice/crio-7207f7498604b578d23a6ae09490522f8c408ebc7ea0ee4041c1d4bca6b3b037 WatchSource:0}: Error finding container 7207f7498604b578d23a6ae09490522f8c408ebc7ea0ee4041c1d4bca6b3b037: Status 404 returned error can't find the container with id 7207f7498604b578d23a6ae09490522f8c408ebc7ea0ee4041c1d4bca6b3b037 Jul 15 22:33:12.022715 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:12.022693 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd15363_1a4e_4956_b662_f8c68f813bd9.slice/crio-dd82cf02f11d5b209759f27dc5993af6538f9b36fed8ede4b10f3698071955de WatchSource:0}: Error finding container dd82cf02f11d5b209759f27dc5993af6538f9b36fed8ede4b10f3698071955de: Status 404 returned error can't find the container with id dd82cf02f11d5b209759f27dc5993af6538f9b36fed8ede4b10f3698071955de Jul 15 22:33:12.023979 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:12.023958 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c753c4f_e47f_4f3f_8fe2_4d41fa627ea9.slice/crio-38351ba0eb2339c3b2e2e60bc04ea422f6cfeaac90f8a6a5063dadb49ee1e7e4 WatchSource:0}: Error finding container 38351ba0eb2339c3b2e2e60bc04ea422f6cfeaac90f8a6a5063dadb49ee1e7e4: Status 404 returned error can't find the container with id 38351ba0eb2339c3b2e2e60bc04ea422f6cfeaac90f8a6a5063dadb49ee1e7e4 Jul 15 22:33:12.185252 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.185202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:12.185452 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.185283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:12.185452 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.185378 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:12.185452 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.185414 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:12.185616 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.185453 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret podName:2a63fe02-5823-4b44-b017-059e1e8ab33d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:13.185436703 +0000 UTC m=+4.125639232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret") pod "global-pull-secret-syncer-gp62w" (UID: "2a63fe02-5823-4b44-b017-059e1e8ab33d") : object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:12.185616 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.185471 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs podName:7b9b6737-f701-4ecb-86f1-0c25f73efdf6 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:13.185464843 +0000 UTC m=+4.125667372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs") pod "network-metrics-daemon-jhmzm" (UID: "7b9b6737-f701-4ecb-86f1-0c25f73efdf6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:12.285946 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.285909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:12.286156 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.286128 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 22:33:12.286156 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.286158 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 22:33:12.286307 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.286190 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gmbkq for pod openshift-network-diagnostics/network-check-target-v26ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:12.286307 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.286261 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq podName:a3257827-435d-40d8-ad19-955d7041302e nodeName:}" failed. No retries permitted until 2026-07-15 22:33:13.286239797 +0000 UTC m=+4.226442341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gmbkq" (UniqueName: "kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq") pod "network-check-target-v26ns" (UID: "a3257827-435d-40d8-ad19-955d7041302e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:12.513786 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.513686 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 22:28:11 +0000 UTC" deadline="2028-01-16 01:31:21.868994622 +0000 UTC" Jul 15 22:33:12.513786 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.513750 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13178h58m9.355249037s" Jul 15 22:33:12.579558 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.579515 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:12.579558 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.579572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:12.579814 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.579671 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:12.579814 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:12.579791 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:12.591511 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.591425 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pk7xr" event={"ID":"27f508ca-8bf5-4bfb-9e27-4f543593132c","Type":"ContainerStarted","Data":"7207f7498604b578d23a6ae09490522f8c408ebc7ea0ee4041c1d4bca6b3b037"} Jul 15 22:33:12.592802 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.592773 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xskg" event={"ID":"1daa7794-3d7e-43b0-b053-6ff4d17f71ec","Type":"ContainerStarted","Data":"5431a571e3798281020255403aaa0513260a85663479a6d1928fd7d7f2688fd0"} Jul 15 22:33:12.594323 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.594287 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"7dab05ad207ab60885980912ed21ebe0a640c1affd8ece0ed5a4ab1bd9626bae"} Jul 15 22:33:12.595653 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.595624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hdrnr" event={"ID":"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3","Type":"ContainerStarted","Data":"5c292c74b1029c214501c40ab7fe68e73fade8267720b44180352074349fb0e8"} Jul 15 22:33:12.596846 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.596825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jfgkz" event={"ID":"5d432250-ab22-419b-b783-26186a2bcfbf","Type":"ContainerStarted","Data":"465e58b139a44f48732cad621e91a68c961e9cc9d34c099c33b0846551b2025c"} Jul 15 22:33:12.597973 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.597953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerStarted","Data":"aecd6a2378bebfd4d802af7ff552188ec272f19bdba0c50be656ba337a116b58"} Jul 15 22:33:12.599164 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.599137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" event={"ID":"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8","Type":"ContainerStarted","Data":"5b294839fbf4cadc294474a54fa528f03d039636eaf3028f5f963fa6f0e88680"} Jul 15 22:33:12.600942 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.600912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"010e8802e63dc57e897d2d31228c5508","Type":"ContainerStarted","Data":"82c8f0c047ec9684ec0dbbcc122ec34efcef2637d7a96a5b7b07bbb926bc1aa1"} Jul 15 22:33:12.601962 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.601943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6vt2l" event={"ID":"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9","Type":"ContainerStarted","Data":"38351ba0eb2339c3b2e2e60bc04ea422f6cfeaac90f8a6a5063dadb49ee1e7e4"} Jul 15 22:33:12.602971 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:12.602953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5kgf2" event={"ID":"1bd15363-1a4e-4956-b662-f8c68f813bd9","Type":"ContainerStarted","Data":"dd82cf02f11d5b209759f27dc5993af6538f9b36fed8ede4b10f3698071955de"} Jul 15 22:33:13.200347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:13.199471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:13.200347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:13.199555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:13.200347 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.199693 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:13.200347 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.199762 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs podName:7b9b6737-f701-4ecb-86f1-0c25f73efdf6 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:15.199740425 +0000 UTC m=+6.139942965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs") pod "network-metrics-daemon-jhmzm" (UID: "7b9b6737-f701-4ecb-86f1-0c25f73efdf6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:13.200347 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.200214 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:13.200347 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.200301 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret podName:2a63fe02-5823-4b44-b017-059e1e8ab33d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:15.200283093 +0000 UTC m=+6.140485631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret") pod "global-pull-secret-syncer-gp62w" (UID: "2a63fe02-5823-4b44-b017-059e1e8ab33d") : object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:13.301573 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:13.300826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:13.301573 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.301013 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 22:33:13.301573 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.301037 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 22:33:13.301573 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.301052 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gmbkq for pod openshift-network-diagnostics/network-check-target-v26ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:13.301573 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.301118 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq podName:a3257827-435d-40d8-ad19-955d7041302e nodeName:}" failed. No retries permitted until 2026-07-15 22:33:15.301097352 +0000 UTC m=+6.241299887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gmbkq" (UniqueName: "kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq") pod "network-check-target-v26ns" (UID: "a3257827-435d-40d8-ad19-955d7041302e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:13.580329 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:13.579034 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:13.580329 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:13.579186 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:13.620294 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:13.619473 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" event={"ID":"433ef3a1b26ec88b78ef4885409e47ef","Type":"ContainerStarted","Data":"d338ebc35ac95b9abebaef8f84576cdad3db5fe25d711f8f592cb2bb525e6b60"} Jul 15 22:33:13.624799 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:13.624758 2565 generic.go:356] "Generic (PLEG): container finished" podID="010e8802e63dc57e897d2d31228c5508" containerID="82c8f0c047ec9684ec0dbbcc122ec34efcef2637d7a96a5b7b07bbb926bc1aa1" exitCode=0 Jul 15 22:33:13.624988 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:13.624820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"010e8802e63dc57e897d2d31228c5508","Type":"ContainerDied","Data":"82c8f0c047ec9684ec0dbbcc122ec34efcef2637d7a96a5b7b07bbb926bc1aa1"} Jul 15 22:33:13.660693 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:13.660594 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-194.ec2.internal" podStartSLOduration=2.660567151 podStartE2EDuration="2.660567151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:13.638454426 +0000 UTC m=+4.578656982" watchObservedRunningTime="2026-07-15 22:33:13.660567151 +0000 UTC m=+4.600769705" Jul 15 22:33:14.579556 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:14.579495 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:14.579781 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:14.579661 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:14.580147 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:14.580093 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:14.580300 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:14.580273 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:14.649789 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:14.649742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" event={"ID":"010e8802e63dc57e897d2d31228c5508","Type":"ContainerStarted","Data":"4a3f70f6c797fd505789637946e68a47e5e2bb7489928d50a999c4bc40b4be3a"} Jul 15 22:33:15.221006 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:15.220966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:15.221274 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:15.221022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:15.221274 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.221184 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:15.221274 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.221238 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs podName:7b9b6737-f701-4ecb-86f1-0c25f73efdf6 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:19.221221915 +0000 UTC m=+10.161424445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs") pod "network-metrics-daemon-jhmzm" (UID: "7b9b6737-f701-4ecb-86f1-0c25f73efdf6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:15.221489 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.221473 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:15.221544 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.221524 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret podName:2a63fe02-5823-4b44-b017-059e1e8ab33d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:19.22150983 +0000 UTC m=+10.161712362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret") pod "global-pull-secret-syncer-gp62w" (UID: "2a63fe02-5823-4b44-b017-059e1e8ab33d") : object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:15.322371 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:15.322328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:15.322588 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.322531 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 22:33:15.322588 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.322560 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 22:33:15.322588 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.322574 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gmbkq for pod openshift-network-diagnostics/network-check-target-v26ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:15.322758 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.322643 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq podName:a3257827-435d-40d8-ad19-955d7041302e nodeName:}" failed. No retries permitted until 2026-07-15 22:33:19.322622652 +0000 UTC m=+10.262825199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gmbkq" (UniqueName: "kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq") pod "network-check-target-v26ns" (UID: "a3257827-435d-40d8-ad19-955d7041302e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:15.582486 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:15.579681 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:15.582486 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:15.581488 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:16.580256 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:16.579615 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:16.580256 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:16.579746 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:16.580256 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:16.580199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:16.580859 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:16.580390 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:17.581493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:17.581454 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:17.581984 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:17.581605 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:18.579755 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:18.579068 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:18.579755 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:18.579115 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:18.579755 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:18.579257 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:18.579755 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:18.579404 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:19.257844 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:19.257805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:19.258376 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:19.257866 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:19.258376 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.257983 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:19.258376 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.257993 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:19.258376 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.258048 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs podName:7b9b6737-f701-4ecb-86f1-0c25f73efdf6 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:27.258033241 +0000 UTC m=+18.198235770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs") pod "network-metrics-daemon-jhmzm" (UID: "7b9b6737-f701-4ecb-86f1-0c25f73efdf6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:19.258376 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.258064 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret podName:2a63fe02-5823-4b44-b017-059e1e8ab33d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:27.25805751 +0000 UTC m=+18.198260040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret") pod "global-pull-secret-syncer-gp62w" (UID: "2a63fe02-5823-4b44-b017-059e1e8ab33d") : object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:19.358387 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:19.358315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:19.358592 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.358507 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 22:33:19.358592 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.358536 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 22:33:19.358592 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.358552 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gmbkq for pod openshift-network-diagnostics/network-check-target-v26ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:19.358755 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.358630 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq podName:a3257827-435d-40d8-ad19-955d7041302e nodeName:}" failed. No retries permitted until 2026-07-15 22:33:27.358607532 +0000 UTC m=+18.298810085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gmbkq" (UniqueName: "kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq") pod "network-check-target-v26ns" (UID: "a3257827-435d-40d8-ad19-955d7041302e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:19.580331 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:19.580225 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:19.580528 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:19.580379 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:20.579240 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:20.579202 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:20.579240 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:20.579224 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:20.579802 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:20.579352 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:20.579802 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:20.579497 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:21.579483 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:21.579447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:21.579948 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:21.579586 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:22.579249 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:22.579205 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:22.579421 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:22.579354 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:22.579763 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:22.579742 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:22.580082 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:22.579865 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:23.578780 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:23.578741 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:23.578976 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:23.578886 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:24.579321 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:24.579285 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:24.579321 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:24.579319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:24.579831 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:24.579433 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:24.579831 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:24.579553 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:25.579498 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:25.579459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:25.579899 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:25.579602 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:26.579289 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:26.579253 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:26.579584 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:26.579253 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:26.579584 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:26.579380 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:26.579584 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:26.579482 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:27.324678 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:27.324640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:27.324872 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:27.324700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:27.324872 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.324806 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:27.324872 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.324807 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:27.324994 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.324876 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs podName:7b9b6737-f701-4ecb-86f1-0c25f73efdf6 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:43.324854106 +0000 UTC m=+34.265056636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs") pod "network-metrics-daemon-jhmzm" (UID: "7b9b6737-f701-4ecb-86f1-0c25f73efdf6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:27.324994 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.324893 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret podName:2a63fe02-5823-4b44-b017-059e1e8ab33d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:43.324886492 +0000 UTC m=+34.265089020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret") pod "global-pull-secret-syncer-gp62w" (UID: "2a63fe02-5823-4b44-b017-059e1e8ab33d") : object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:27.425457 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:27.425416 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:27.425633 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.425609 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 22:33:27.425680 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.425637 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 22:33:27.425680 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.425651 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gmbkq for pod openshift-network-diagnostics/network-check-target-v26ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:27.425755 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.425718 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq podName:a3257827-435d-40d8-ad19-955d7041302e nodeName:}" failed. No retries permitted until 2026-07-15 22:33:43.425700615 +0000 UTC m=+34.365903164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gmbkq" (UniqueName: "kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq") pod "network-check-target-v26ns" (UID: "a3257827-435d-40d8-ad19-955d7041302e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:27.579681 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:27.579598 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:27.580099 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:27.579712 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:28.579243 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:28.579209 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:28.579445 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:28.579256 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:28.579445 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:28.579355 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:28.579561 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:28.579504 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:29.579584 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:29.579542 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:29.579962 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:29.579646 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:30.578921 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:30.578884 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:30.578921 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:30.578906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:30.579144 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:30.579010 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:30.579219 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:30.579143 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:31.579441 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:31.579400 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:31.579975 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:31.579541 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:32.579607 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.579008 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:32.579607 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:32.579367 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:32.579607 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.579455 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:32.579607 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:32.579542 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:32.692927 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.692675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5kgf2" event={"ID":"1bd15363-1a4e-4956-b662-f8c68f813bd9","Type":"ContainerStarted","Data":"ef6aee7402cac44f4209217d5def1e054322b384db17fb05c342aa724b0ba80f"} Jul 15 22:33:32.694397 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.694365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pk7xr" event={"ID":"27f508ca-8bf5-4bfb-9e27-4f543593132c","Type":"ContainerStarted","Data":"249b0f3640ee41901844290eb4ff637b1b6fd0f226c7e68b363e93301354722c"} Jul 15 22:33:32.696052 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.696014 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xskg" event={"ID":"1daa7794-3d7e-43b0-b053-6ff4d17f71ec","Type":"ContainerStarted","Data":"4ddd1a7aa009839342521fe46e856045fc6b3d74f11085f3c1e6c9fa4464beb3"} Jul 15 22:33:32.697502 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.697479 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"b5bcbd055a5ce0a1958e63681fbcc2623a491d16c269a53a5d038d6f8a50ef6b"} Jul 15 22:33:32.699017 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.698994 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hdrnr" event={"ID":"1cf1c5d5-ba93-4ef4-b043-8233b97b14d3","Type":"ContainerStarted","Data":"b3c32a678726319498992efca36e02e5ff69684d553fb326889f3b5363a3d8e2"} Jul 15 22:33:32.700585 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.700562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jfgkz" event={"ID":"5d432250-ab22-419b-b783-26186a2bcfbf","Type":"ContainerStarted","Data":"93f0264a00c4cfd4eb11e1b3e13859ac18330a0e1fa0e05e56f62ddca2807246"} Jul 15 22:33:32.701997 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.701977 2565 generic.go:356] "Generic (PLEG): container finished" podID="a449bf70-dfdc-4c4a-9248-ae820d7dbe71" containerID="fd094af559555145b04524b06d5e5b604823baad0882746529f436a9510cf24b" exitCode=0 Jul 15 22:33:32.702114 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.702043 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerDied","Data":"fd094af559555145b04524b06d5e5b604823baad0882746529f436a9510cf24b"} Jul 15 22:33:32.704152 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.703930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" event={"ID":"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8","Type":"ContainerStarted","Data":"efeb5ef4fe04c685cb672e2dc3e31c3c1c568ae1748d4437b41edda1ed3a312a"} Jul 15 22:33:32.711438 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.711398 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5kgf2" podStartSLOduration=2.412765172 podStartE2EDuration="22.711366505s" totalImagesPullingTime="20.298601333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:32.711343573 +0000 UTC m=+23.651546124" watchObservedRunningTime="2026-07-15 22:33:32.711366505 +0000 UTC m=+23.651569057" Jul 15 22:33:32.711733 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.711704 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-194.ec2.internal" podStartSLOduration=21.711694524 podStartE2EDuration="21.711694524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:14.668331975 +0000 UTC m=+5.608534529" watchObservedRunningTime="2026-07-15 22:33:32.711694524 +0000 UTC m=+23.651897075" Jul 15 22:33:32.748852 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.748802 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pk7xr" podStartSLOduration=2.344845059 podStartE2EDuration="22.748787548s" totalImagesPullingTime="20.403942489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:32.727923729 +0000 UTC m=+23.668126280" watchObservedRunningTime="2026-07-15 22:33:32.748787548 +0000 UTC m=+23.688990097" Jul 15 22:33:32.766784 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.766733 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7xskg" podStartSLOduration=2.408126163 podStartE2EDuration="22.766716651s" totalImagesPullingTime="20.358590488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:32.76643764 +0000 UTC m=+23.706640192" watchObservedRunningTime="2026-07-15 22:33:32.766716651 +0000 UTC m=+23.706919200" Jul 15 22:33:32.781856 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.781811 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hdrnr" podStartSLOduration=2.43335845 podStartE2EDuration="22.78179577s" totalImagesPullingTime="20.34843732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:32.78160429 +0000 UTC m=+23.721806834" watchObservedRunningTime="2026-07-15 22:33:32.78179577 +0000 UTC m=+23.721998319" Jul 15 22:33:32.798465 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:32.798425 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jfgkz" podStartSLOduration=2.440680433 podStartE2EDuration="22.798406689s" totalImagesPullingTime="20.357726256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:32.798030338 +0000 UTC m=+23.738232888" watchObservedRunningTime="2026-07-15 22:33:32.798406689 +0000 UTC m=+23.738609238" Jul 15 22:33:33.579794 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:33.579758 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:33.580351 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:33.579903 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:33.709692 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:33.709653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"3dc4b80fc1f8cafd973c0f85381fc601e575c30b054233a55fb03016c3c55a48"} Jul 15 22:33:33.709875 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:33.709754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"2559ec71dd802f8e3872ec1702981e8ebf3cea1d76c67f96158eabe2a1c90343"} Jul 15 22:33:33.709875 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:33.709809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"02afde478387ab858bdef55b0094dbc45ff79fedfd13796c0e6bc66a10a07d49"} Jul 15 22:33:33.709875 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:33.709855 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"e49355945ef0fb5d5bfac7c0edaf3fb2962a82840fa901bf49d25a2dec807abc"} Jul 15 22:33:33.710005 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:33.709900 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"7c6a31f37e37f436ecccd6ecd16fbb2f068494802068a4bd4822a2a593bee232"} Jul 15 22:33:33.972415 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:33.972205 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 22:33:34.533822 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:34.533724 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T22:33:33.972408915Z","UUID":"3bafebdb-9eaa-4acc-a131-26be4fbb666e","Handler":null,"Name":"","Endpoint":""} Jul 15 22:33:34.535399 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:34.535376 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 22:33:34.535541 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:34.535423 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 22:33:34.580182 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:34.580141 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:34.580182 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:34.580177 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:34.580623 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:34.580288 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:34.580712 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:34.580676 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:34.715150 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:34.715107 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" event={"ID":"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8","Type":"ContainerStarted","Data":"aca96a36cb77c1cbe8e40978eeb89a0ccd1d9c1cf89f2b948b16f5e110dca2fd"} Jul 15 22:33:34.716939 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:34.716912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6vt2l" event={"ID":"2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9","Type":"ContainerStarted","Data":"107ee27439b17d10830b6e488fc2c51715fcf849ef8b3f28ddeb3303d5a6ae0e"} Jul 15 22:33:34.734284 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:34.734238 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6vt2l" podStartSLOduration=4.378408956 podStartE2EDuration="24.734221752s" totalImagesPullingTime="20.355812796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:34.733762361 +0000 UTC m=+25.673964912" watchObservedRunningTime="2026-07-15 22:33:34.734221752 +0000 UTC m=+25.674424306" Jul 15 22:33:35.578804 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:35.578760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:35.579048 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:35.578910 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:35.722684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:35.722646 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"cd46de4df557a14509cf4c98b3dfd9646ef41252fb8ce78d9a6aea97255b651d"} Jul 15 22:33:36.346034 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:36.346004 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:36.346699 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:36.346674 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:36.579467 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:36.579427 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:36.579662 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:36.579441 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:36.579662 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:36.579588 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:36.579844 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:36.579673 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:36.727235 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:36.727206 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:36.727845 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:36.727696 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5kgf2" Jul 15 22:33:37.579558 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.579454 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:37.579704 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:37.579568 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:37.732482 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.732442 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" event={"ID":"7c4945d4-04f4-480b-9abb-5db6c5a8a87c","Type":"ContainerStarted","Data":"8cad7919167286c63e4aa753fdb18b1ef9619d6abdc8fa84aacdbce510a6be6f"} Jul 15 22:33:37.733052 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.732682 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:37.734155 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.734130 2565 generic.go:356] "Generic (PLEG): container finished" podID="a449bf70-dfdc-4c4a-9248-ae820d7dbe71" containerID="2d2ecd7fcf8cea4234be608d867f88f33e08e5d07134d5c4e5608707379423ce" exitCode=0 Jul 15 22:33:37.734284 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.734196 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerDied","Data":"2d2ecd7fcf8cea4234be608d867f88f33e08e5d07134d5c4e5608707379423ce"} Jul 15 22:33:37.736423 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.736394 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" event={"ID":"b4abc8f6-aea5-43f8-b461-f5f7bf9528d8","Type":"ContainerStarted","Data":"1136baac34f455a638eb1a9e7f09ee7b1c7eedcda5561d07ce6af4c6d99a2ec3"} Jul 15 22:33:37.749672 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.749653 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:37.767476 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.767434 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" podStartSLOduration=7.336507606 podStartE2EDuration="27.767420735s" totalImagesPullingTime="20.430913129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:37.766807124 +0000 UTC m=+28.707009675" watchObservedRunningTime="2026-07-15 22:33:37.767420735 +0000 UTC m=+28.707623285" Jul 15 22:33:37.818148 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:37.818099 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rkjj" podStartSLOduration=3.363950406 podStartE2EDuration="27.818084093s" totalImagesPullingTime="24.454133687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:37.785852592 +0000 UTC m=+28.726055142" watchObservedRunningTime="2026-07-15 22:33:37.818084093 +0000 UTC m=+28.758286643" Jul 15 22:33:38.579287 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:38.579240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:38.579508 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:38.579387 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:38.579851 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:38.579826 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:38.579947 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:38.579926 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:38.739716 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:38.739674 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:38.740304 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:38.739740 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:38.754941 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:38.754914 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:33:39.257900 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:39.257707 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gp62w"] Jul 15 22:33:39.258091 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:39.258000 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:39.258158 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:39.258111 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:39.260766 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:39.260740 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v26ns"] Jul 15 22:33:39.260880 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:39.260845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:39.260944 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:39.260927 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:39.273030 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:39.273003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhmzm"] Jul 15 22:33:39.273133 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:39.273101 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:39.273249 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:39.273205 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:39.743629 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:39.743595 2565 generic.go:356] "Generic (PLEG): container finished" podID="a449bf70-dfdc-4c4a-9248-ae820d7dbe71" containerID="112123595be08044b1120a98929020e53eaf16227718f27c38f7806ca54f423a" exitCode=0 Jul 15 22:33:39.744305 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:39.743689 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerDied","Data":"112123595be08044b1120a98929020e53eaf16227718f27c38f7806ca54f423a"} Jul 15 22:33:40.579307 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:40.579221 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:40.579485 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:40.579238 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:40.579485 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:40.579345 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:40.579485 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:40.579241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:40.579485 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:40.579409 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:40.579631 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:40.579499 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:40.749223 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:40.749194 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerStarted","Data":"51f675cd1a29b38fecb2ef1502a99cab717e3f1a11df370a883a4d18160e2999"} Jul 15 22:33:41.753385 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:41.753354 2565 generic.go:356] "Generic (PLEG): container finished" podID="a449bf70-dfdc-4c4a-9248-ae820d7dbe71" containerID="51f675cd1a29b38fecb2ef1502a99cab717e3f1a11df370a883a4d18160e2999" exitCode=0 Jul 15 22:33:41.753836 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:41.753399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerDied","Data":"51f675cd1a29b38fecb2ef1502a99cab717e3f1a11df370a883a4d18160e2999"} Jul 15 22:33:42.579062 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:42.579026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:42.579062 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:42.579059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:42.579324 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:42.579026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:42.579324 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:42.579160 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:42.579324 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:42.579235 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:42.579324 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:42.579305 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:43.344664 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:43.344630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:43.345339 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:43.344681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:43.345339 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:43.344803 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:43.345339 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:43.344808 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:43.345339 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:43.344863 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs podName:7b9b6737-f701-4ecb-86f1-0c25f73efdf6 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:15.34484715 +0000 UTC m=+66.285049679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs") pod "network-metrics-daemon-jhmzm" (UID: "7b9b6737-f701-4ecb-86f1-0c25f73efdf6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 22:33:43.345339 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:43.344877 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret podName:2a63fe02-5823-4b44-b017-059e1e8ab33d nodeName:}" failed. No retries permitted until 2026-07-15 22:34:15.344871293 +0000 UTC m=+66.285073821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret") pod "global-pull-secret-syncer-gp62w" (UID: "2a63fe02-5823-4b44-b017-059e1e8ab33d") : object "kube-system"/"original-pull-secret" not registered Jul 15 22:33:43.446051 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:43.446014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:43.446248 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:43.446204 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 22:33:43.446248 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:43.446227 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 22:33:43.446248 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:43.446237 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gmbkq for pod openshift-network-diagnostics/network-check-target-v26ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:43.446370 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:43.446287 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq podName:a3257827-435d-40d8-ad19-955d7041302e nodeName:}" failed. No retries permitted until 2026-07-15 22:34:15.446273935 +0000 UTC m=+66.386476464 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gmbkq" (UniqueName: "kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq") pod "network-check-target-v26ns" (UID: "a3257827-435d-40d8-ad19-955d7041302e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 22:33:44.579612 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:44.579576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:44.580074 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:44.579621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:44.580074 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:44.579701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:44.580074 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:44.579720 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhmzm" podUID="7b9b6737-f701-4ecb-86f1-0c25f73efdf6" Jul 15 22:33:44.580074 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:44.579786 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v26ns" podUID="a3257827-435d-40d8-ad19-955d7041302e" Jul 15 22:33:44.580074 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:44.579851 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gp62w" podUID="2a63fe02-5823-4b44-b017-059e1e8ab33d" Jul 15 22:33:45.365475 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:45.365434 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:45.475189 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:45.474964 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:46.305563 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.305531 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-194.ec2.internal" event="NodeReady" Jul 15 22:33:46.305899 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.305661 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 22:33:46.357793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.357753 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fc5875b8d-qlg76"] Jul 15 22:33:46.360763 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.360744 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rsk5w"] Jul 15 22:33:46.360886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.360858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.363487 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.363463 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7pd8h\"" Jul 15 22:33:46.363644 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.363475 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 22:33:46.363710 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.363654 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h77p9"] Jul 15 22:33:46.363796 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.363777 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 22:33:46.363898 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.363786 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.364011 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.363996 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 22:33:46.366384 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.366365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h77p9" Jul 15 22:33:46.368636 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.368620 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 22:33:46.368736 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.368718 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 22:33:46.368808 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.368792 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-trnq4\"" Jul 15 22:33:46.369684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.369129 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 22:33:46.369684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.369150 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l8qc7\"" Jul 15 22:33:46.369684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.369151 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 22:33:46.369684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.369449 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 22:33:46.370136 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.370118 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 22:33:46.374485 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.374468 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h77p9"] Jul 15 22:33:46.376475 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.376451 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rsk5w"] Jul 15 22:33:46.379720 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.379699 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fc5875b8d-qlg76"] Jul 15 22:33:46.468688 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.468688 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-config-volume\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.468688 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468645 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv577\" (UniqueName: \"kubernetes.io/projected/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-kube-api-access-lv577\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.468688 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.468948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28a47249-51b6-40ed-8050-0715614d7a82-ca-trust-extracted\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.468948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-registry-certificates\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.468948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-image-registry-private-configuration\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.468948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-bound-sa-token\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.468948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acf3b5e8-bdb1-4831-a9b3-6febe54464cd-cert\") pod \"ingress-canary-h77p9\" (UID: \"acf3b5e8-bdb1-4831-a9b3-6febe54464cd\") " pod="openshift-ingress-canary/ingress-canary-h77p9" Jul 15 22:33:46.468948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jhd\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-kube-api-access-h4jhd\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.468948 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-trusted-ca\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.469245 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvzk\" (UniqueName: \"kubernetes.io/projected/acf3b5e8-bdb1-4831-a9b3-6febe54464cd-kube-api-access-nfvzk\") pod \"ingress-canary-h77p9\" (UID: \"acf3b5e8-bdb1-4831-a9b3-6febe54464cd\") " pod="openshift-ingress-canary/ingress-canary-h77p9" Jul 15 22:33:46.469245 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.468983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-tmp-dir\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.469245 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.469034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-installation-pull-secrets\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.504231 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.504189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:46.569553 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569514 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-installation-pull-secrets\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.569553 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.569766 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-config-volume\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.569766 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv577\" (UniqueName: \"kubernetes.io/projected/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-kube-api-access-lv577\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.569766 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.569766 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:46.569691 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 22:33:46.569766 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:46.569688 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 22:33:46.569766 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:46.569703 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc5875b8d-qlg76: secret "image-registry-tls" not found Jul 15 22:33:46.569766 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:46.569763 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls podName:bd6596a2-25e0-4482-bd65-38fb27ddfe8d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:47.069742301 +0000 UTC m=+38.009944832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls") pod "dns-default-rsk5w" (UID: "bd6596a2-25e0-4482-bd65-38fb27ddfe8d") : secret "dns-default-metrics-tls" not found Jul 15 22:33:46.570046 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:46.569821 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls podName:28a47249-51b6-40ed-8050-0715614d7a82 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:47.069791253 +0000 UTC m=+38.009993787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls") pod "image-registry-fc5875b8d-qlg76" (UID: "28a47249-51b6-40ed-8050-0715614d7a82") : secret "image-registry-tls" not found Jul 15 22:33:46.570046 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28a47249-51b6-40ed-8050-0715614d7a82-ca-trust-extracted\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.570046 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-registry-certificates\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.570046 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-image-registry-private-configuration\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.570046 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.569952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-bound-sa-token\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.570322 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acf3b5e8-bdb1-4831-a9b3-6febe54464cd-cert\") pod \"ingress-canary-h77p9\" (UID: \"acf3b5e8-bdb1-4831-a9b3-6febe54464cd\") " pod="openshift-ingress-canary/ingress-canary-h77p9" Jul 15 22:33:46.570385 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570358 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28a47249-51b6-40ed-8050-0715614d7a82-ca-trust-extracted\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.570385 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-config-volume\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.570481 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jhd\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-kube-api-access-h4jhd\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.570530 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-trusted-ca\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.570530 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvzk\" (UniqueName: \"kubernetes.io/projected/acf3b5e8-bdb1-4831-a9b3-6febe54464cd-kube-api-access-nfvzk\") pod \"ingress-canary-h77p9\" (UID: \"acf3b5e8-bdb1-4831-a9b3-6febe54464cd\") " pod="openshift-ingress-canary/ingress-canary-h77p9" Jul 15 22:33:46.570616 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-tmp-dir\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.570616 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-registry-certificates\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.571125 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.570912 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-tmp-dir\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.571936 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.571916 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-trusted-ca\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.574240 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.574218 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-image-registry-private-configuration\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.574310 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.574218 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-installation-pull-secrets\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.574310 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.574224 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acf3b5e8-bdb1-4831-a9b3-6febe54464cd-cert\") pod \"ingress-canary-h77p9\" (UID: \"acf3b5e8-bdb1-4831-a9b3-6febe54464cd\") " pod="openshift-ingress-canary/ingress-canary-h77p9" Jul 15 22:33:46.578753 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.578700 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:33:46.579684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.579131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:33:46.579684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.579296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvzk\" (UniqueName: \"kubernetes.io/projected/acf3b5e8-bdb1-4831-a9b3-6febe54464cd-kube-api-access-nfvzk\") pod \"ingress-canary-h77p9\" (UID: \"acf3b5e8-bdb1-4831-a9b3-6febe54464cd\") " pod="openshift-ingress-canary/ingress-canary-h77p9" Jul 15 22:33:46.579684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.579325 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jhd\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-kube-api-access-h4jhd\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.579886 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.579686 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:33:46.580321 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.580303 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv577\" (UniqueName: \"kubernetes.io/projected/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-kube-api-access-lv577\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:46.580427 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.580408 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-bound-sa-token\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:46.581772 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.581754 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 22:33:46.581870 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.581783 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-84k7s\"" Jul 15 22:33:46.581870 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.581788 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 22:33:46.582203 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.582188 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 22:33:46.582250 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.582230 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 22:33:46.582250 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.582236 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmq28\"" Jul 15 22:33:46.604888 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.604863 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:46.684491 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.684451 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h77p9" Jul 15 22:33:46.768860 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.768831 2565 generic.go:356] "Generic (PLEG): container finished" podID="a449bf70-dfdc-4c4a-9248-ae820d7dbe71" containerID="8a5b5fdc94ebd6571a6d4879e8ca254dd3aac3b667202b85675f37fe3f3a699c" exitCode=0 Jul 15 22:33:46.769007 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.768876 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerDied","Data":"8a5b5fdc94ebd6571a6d4879e8ca254dd3aac3b667202b85675f37fe3f3a699c"} Jul 15 22:33:46.828082 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:46.828053 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h77p9"] Jul 15 22:33:46.838284 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:46.838256 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf3b5e8_bdb1_4831_a9b3_6febe54464cd.slice/crio-8644074d858ff906b1e23737f05840625d3fc0fb41f1ab6f744bea51e9029041 WatchSource:0}: Error finding container 8644074d858ff906b1e23737f05840625d3fc0fb41f1ab6f744bea51e9029041: Status 404 returned error can't find the container with id 8644074d858ff906b1e23737f05840625d3fc0fb41f1ab6f744bea51e9029041 Jul 15 22:33:47.074527 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:47.074288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:47.074670 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:47.074434 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 22:33:47.074670 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:47.074551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:47.074670 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:47.074559 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc5875b8d-qlg76: secret "image-registry-tls" not found Jul 15 22:33:47.074670 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:47.074608 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls podName:28a47249-51b6-40ed-8050-0715614d7a82 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:48.074593413 +0000 UTC m=+39.014795941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls") pod "image-registry-fc5875b8d-qlg76" (UID: "28a47249-51b6-40ed-8050-0715614d7a82") : secret "image-registry-tls" not found Jul 15 22:33:47.074670 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:47.074641 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 22:33:47.074835 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:47.074689 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls podName:bd6596a2-25e0-4482-bd65-38fb27ddfe8d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:48.074677776 +0000 UTC m=+39.014880304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls") pod "dns-default-rsk5w" (UID: "bd6596a2-25e0-4482-bd65-38fb27ddfe8d") : secret "dns-default-metrics-tls" not found Jul 15 22:33:47.775066 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:47.775032 2565 generic.go:356] "Generic (PLEG): container finished" podID="a449bf70-dfdc-4c4a-9248-ae820d7dbe71" containerID="8aedd16e461c182dc61168db0d3a398e51473324f5cdc23a244d3c682b85681b" exitCode=0 Jul 15 22:33:47.775556 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:47.775098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerDied","Data":"8aedd16e461c182dc61168db0d3a398e51473324f5cdc23a244d3c682b85681b"} Jul 15 22:33:47.777463 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:47.777429 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h77p9" event={"ID":"acf3b5e8-bdb1-4831-a9b3-6febe54464cd","Type":"ContainerStarted","Data":"8644074d858ff906b1e23737f05840625d3fc0fb41f1ab6f744bea51e9029041"} Jul 15 22:33:48.084217 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:48.084124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:48.084217 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:48.084197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:48.084435 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:48.084297 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 22:33:48.084435 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:48.084360 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 22:33:48.084435 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:48.084381 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc5875b8d-qlg76: secret "image-registry-tls" not found Jul 15 22:33:48.084435 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:48.084370 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls podName:bd6596a2-25e0-4482-bd65-38fb27ddfe8d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:50.084351709 +0000 UTC m=+41.024554257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls") pod "dns-default-rsk5w" (UID: "bd6596a2-25e0-4482-bd65-38fb27ddfe8d") : secret "dns-default-metrics-tls" not found Jul 15 22:33:48.084590 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:48.084444 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls podName:28a47249-51b6-40ed-8050-0715614d7a82 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:50.084428031 +0000 UTC m=+41.024630560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls") pod "image-registry-fc5875b8d-qlg76" (UID: "28a47249-51b6-40ed-8050-0715614d7a82") : secret "image-registry-tls" not found Jul 15 22:33:48.783425 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:48.783389 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6bx82" event={"ID":"a449bf70-dfdc-4c4a-9248-ae820d7dbe71","Type":"ContainerStarted","Data":"cd511f11520cbb8a34a2733c0a4fc983dbb3d243bce9077b2c08016631887e55"} Jul 15 22:33:48.805964 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:48.805913 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6bx82" podStartSLOduration=7.819923708 podStartE2EDuration="38.805897639s" totalImagesPullingTime="30.985973931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:48.805449434 +0000 UTC m=+39.745651984" watchObservedRunningTime="2026-07-15 22:33:48.805897639 +0000 UTC m=+39.746100201" Jul 15 22:33:48.943829 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:48.943793 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:49.003289 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:49.003256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:50.103676 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.103637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:50.104270 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.103698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:50.104270 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:50.103800 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 22:33:50.104270 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:50.103864 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 22:33:50.104270 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:50.103886 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc5875b8d-qlg76: secret "image-registry-tls" not found Jul 15 22:33:50.104270 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:50.103873 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls podName:bd6596a2-25e0-4482-bd65-38fb27ddfe8d nodeName:}" failed. No retries permitted until 2026-07-15 22:33:54.103851799 +0000 UTC m=+45.044054327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls") pod "dns-default-rsk5w" (UID: "bd6596a2-25e0-4482-bd65-38fb27ddfe8d") : secret "dns-default-metrics-tls" not found Jul 15 22:33:50.104270 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:50.103957 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls podName:28a47249-51b6-40ed-8050-0715614d7a82 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:54.103938018 +0000 UTC m=+45.044140552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls") pod "image-registry-fc5875b8d-qlg76" (UID: "28a47249-51b6-40ed-8050-0715614d7a82") : secret "image-registry-tls" not found Jul 15 22:33:50.123556 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.123528 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:50.306816 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.306729 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:50.370715 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.370682 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:50.434148 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.434092 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:50.782746 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.782711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:50.789741 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.789713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h77p9" event={"ID":"acf3b5e8-bdb1-4831-a9b3-6febe54464cd","Type":"ContainerStarted","Data":"8dae79acaf14a7fec89a7e4bef41a3429af1b1e94209d99a754915548e969568"} Jul 15 22:33:50.806391 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.806340 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h77p9" podStartSLOduration=1.757996339 podStartE2EDuration="4.806324399s" totalImagesPullingTime="3.04832806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:50.805719286 +0000 UTC m=+41.745921836" watchObservedRunningTime="2026-07-15 22:33:50.806324399 +0000 UTC m=+41.746526990" Jul 15 22:33:50.880941 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:50.880901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:51.477256 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:51.477211 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:51.538350 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:51.538314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:51.732552 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:51.732457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:51.833231 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:51.833200 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:51.932916 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:51.932866 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:53.591254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:53.591220 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:54.134427 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:54.134388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:33:54.134633 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:54.134433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:33:54.134633 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:54.134542 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 22:33:54.134633 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:54.134558 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 22:33:54.134633 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:54.134572 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc5875b8d-qlg76: secret "image-registry-tls" not found Jul 15 22:33:54.134633 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:54.134620 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls podName:bd6596a2-25e0-4482-bd65-38fb27ddfe8d nodeName:}" failed. No retries permitted until 2026-07-15 22:34:02.134604443 +0000 UTC m=+53.074806970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls") pod "dns-default-rsk5w" (UID: "bd6596a2-25e0-4482-bd65-38fb27ddfe8d") : secret "dns-default-metrics-tls" not found Jul 15 22:33:54.134633 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:54.134635 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls podName:28a47249-51b6-40ed-8050-0715614d7a82 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:02.134629223 +0000 UTC m=+53.074831750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls") pod "image-registry-fc5875b8d-qlg76" (UID: "28a47249-51b6-40ed-8050-0715614d7a82") : secret "image-registry-tls" not found Jul 15 22:33:55.305440 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:55.305406 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:55.406316 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:55.406278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:56.585230 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.585198 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r4tsd"] Jul 15 22:33:56.623770 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.623736 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:56.624933 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.624916 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4tsd"] Jul 15 22:33:56.625053 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.625039 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.627606 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.627577 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 22:33:56.627757 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.627577 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 22:33:56.628917 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.628885 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 22:33:56.628917 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.628885 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkr96\"" Jul 15 22:33:56.629064 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.628940 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 22:33:56.651368 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.651340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9bfad52c-9805-44c2-b8e0-5ae8243987d0-crio-socket\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.651368 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.651370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.651544 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.651391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrmm\" (UniqueName: \"kubernetes.io/projected/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-api-access-tkrmm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.651544 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.651449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bfad52c-9805-44c2-b8e0-5ae8243987d0-data-volume\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.651544 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.651468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.715697 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.715662 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:56.752897 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.752867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9bfad52c-9805-44c2-b8e0-5ae8243987d0-crio-socket\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.752897 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.752898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.753099 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.752920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrmm\" (UniqueName: \"kubernetes.io/projected/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-api-access-tkrmm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.753099 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.752991 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9bfad52c-9805-44c2-b8e0-5ae8243987d0-crio-socket\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.753099 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:56.753018 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 22:33:56.753099 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.753048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bfad52c-9805-44c2-b8e0-5ae8243987d0-data-volume\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.753099 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:56.753096 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:57.253074799 +0000 UTC m=+48.193277367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : configmap "kube-rbac-proxy" not found Jul 15 22:33:56.753378 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.753120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.753378 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:56.753307 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 22:33:56.753378 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:56.753363 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:57.253348679 +0000 UTC m=+48.193551225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : secret "insights-runtime-extractor-tls" not found Jul 15 22:33:56.753378 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.753363 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bfad52c-9805-44c2-b8e0-5ae8243987d0-data-volume\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:56.768639 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:56.768620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrmm\" (UniqueName: \"kubernetes.io/projected/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-api-access-tkrmm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:57.256707 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.256667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:57.256875 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.256759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:57.256875 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:57.256858 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 22:33:57.256951 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:57.256866 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 22:33:57.256951 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:57.256940 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:58.256926238 +0000 UTC m=+49.197128766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : configmap "kube-rbac-proxy" not found Jul 15 22:33:57.257017 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:57.256953 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:33:58.256946934 +0000 UTC m=+49.197149461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : secret "insights-runtime-extractor-tls" not found Jul 15 22:33:57.280723 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.280696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:57.287790 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.287765 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4"] Jul 15 22:33:57.321782 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.321747 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4"] Jul 15 22:33:57.321912 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.321866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" Jul 15 22:33:57.324555 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.324525 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 15 22:33:57.325860 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.325825 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 15 22:33:57.325990 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.325874 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-c5qt8\"" Jul 15 22:33:57.346645 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.346619 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:57.357770 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.357747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwjp\" (UniqueName: \"kubernetes.io/projected/0225982a-bd96-4dff-b3f3-a960b4a53d14-kube-api-access-sjwjp\") pod \"migrator-7f59b54f6c-hrns4\" (UID: \"0225982a-bd96-4dff-b3f3-a960b4a53d14\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" Jul 15 22:33:57.412714 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.412685 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:57.459181 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.459123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwjp\" (UniqueName: \"kubernetes.io/projected/0225982a-bd96-4dff-b3f3-a960b4a53d14-kube-api-access-sjwjp\") pod \"migrator-7f59b54f6c-hrns4\" (UID: \"0225982a-bd96-4dff-b3f3-a960b4a53d14\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" Jul 15 22:33:57.467666 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.467637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwjp\" (UniqueName: \"kubernetes.io/projected/0225982a-bd96-4dff-b3f3-a960b4a53d14-kube-api-access-sjwjp\") pod \"migrator-7f59b54f6c-hrns4\" (UID: \"0225982a-bd96-4dff-b3f3-a960b4a53d14\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" Jul 15 22:33:57.512825 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.512748 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:57.631055 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.631019 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" Jul 15 22:33:57.766534 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.766462 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4"] Jul 15 22:33:57.769569 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:33:57.769538 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0225982a_bd96_4dff_b3f3_a960b4a53d14.slice/crio-68ad6d1bd2b720817626c00e8d5c8b4bebcd7ca64f6bbf42579ba92cb477a8ea WatchSource:0}: Error finding container 68ad6d1bd2b720817626c00e8d5c8b4bebcd7ca64f6bbf42579ba92cb477a8ea: Status 404 returned error can't find the container with id 68ad6d1bd2b720817626c00e8d5c8b4bebcd7ca64f6bbf42579ba92cb477a8ea Jul 15 22:33:57.808233 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.808193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" event={"ID":"0225982a-bd96-4dff-b3f3-a960b4a53d14","Type":"ContainerStarted","Data":"68ad6d1bd2b720817626c00e8d5c8b4bebcd7ca64f6bbf42579ba92cb477a8ea"} Jul 15 22:33:57.856060 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.856025 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:57.955986 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:57.955949 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:58.124262 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:58.124176 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:58.186064 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:58.186036 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hdrnr_1cf1c5d5-ba93-4ef4-b043-8233b97b14d3/dns-node-resolver/0.log" Jul 15 22:33:58.223286 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:58.223256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:58.266577 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:58.266536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:58.266734 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:58.266590 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:33:58.266734 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:58.266655 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 22:33:58.266734 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:58.266706 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 22:33:58.266836 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:58.266746 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:00.266729675 +0000 UTC m=+51.206932203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : configmap "kube-rbac-proxy" not found Jul 15 22:33:58.266836 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:33:58.266761 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:00.266754523 +0000 UTC m=+51.206957051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : secret "insights-runtime-extractor-tls" not found Jul 15 22:33:59.185596 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.185561 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jfgkz_5d432250-ab22-419b-b783-26186a2bcfbf/node-ca/0.log" Jul 15 22:33:59.727554 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.727464 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-w8wx9"] Jul 15 22:33:59.750064 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.750032 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-w8wx9"] Jul 15 22:33:59.750238 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.750181 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.753089 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.753067 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 22:33:59.753089 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.753077 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 22:33:59.754138 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.754105 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 22:33:59.754138 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.754106 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 22:33:59.754326 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.754104 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gp99z\"" Jul 15 22:33:59.766607 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.766582 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:59.779180 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.779148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d8e7823-5966-41cb-8a72-79ba54b7f74d-signing-cabundle\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.779276 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.779204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfbr\" (UniqueName: \"kubernetes.io/projected/9d8e7823-5966-41cb-8a72-79ba54b7f74d-kube-api-access-lmfbr\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.779276 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.779252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d8e7823-5966-41cb-8a72-79ba54b7f74d-signing-key\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.787026 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.787002 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h77p9_acf3b5e8-bdb1-4831-a9b3-6febe54464cd/serve-healthcheck-canary/0.log" Jul 15 22:33:59.814716 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.814685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" event={"ID":"0225982a-bd96-4dff-b3f3-a960b4a53d14","Type":"ContainerStarted","Data":"51728dec111206c0ada088d97188de5fbfabdabe3adbe98ffa9c4d9e5c94be90"} Jul 15 22:33:59.814840 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.814744 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" event={"ID":"0225982a-bd96-4dff-b3f3-a960b4a53d14","Type":"ContainerStarted","Data":"7a782729e9e03f9f0587d3fa5152d0766d10548394c893cb8733614b3c8cdded"} Jul 15 22:33:59.830153 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.830091 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hrns4" podStartSLOduration=1.279487749 podStartE2EDuration="2.830070842s" totalImagesPullingTime="1.550583093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:33:59.829931311 +0000 UTC m=+50.770133863" watchObservedRunningTime="2026-07-15 22:33:59.830070842 +0000 UTC m=+50.770273392" Jul 15 22:33:59.880660 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.880607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfbr\" (UniqueName: \"kubernetes.io/projected/9d8e7823-5966-41cb-8a72-79ba54b7f74d-kube-api-access-lmfbr\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.880853 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.880683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d8e7823-5966-41cb-8a72-79ba54b7f74d-signing-key\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.880853 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.880815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d8e7823-5966-41cb-8a72-79ba54b7f74d-signing-cabundle\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.881541 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.881518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d8e7823-5966-41cb-8a72-79ba54b7f74d-signing-cabundle\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.883326 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.883307 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d8e7823-5966-41cb-8a72-79ba54b7f74d-signing-key\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.889497 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.889475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfbr\" (UniqueName: \"kubernetes.io/projected/9d8e7823-5966-41cb-8a72-79ba54b7f74d-kube-api-access-lmfbr\") pod \"service-ca-79f68b9f4f-w8wx9\" (UID: \"9d8e7823-5966-41cb-8a72-79ba54b7f74d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:33:59.910907 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.910874 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:33:59.968073 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:33:59.968041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:00.059265 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:00.059143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" Jul 15 22:34:00.196682 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:00.196648 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-w8wx9"] Jul 15 22:34:00.199948 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:00.199914 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8e7823_5966_41cb_8a72_79ba54b7f74d.slice/crio-79352a93e604a566c5dbb7bf9b674c64a2bd656beca9927e5f826c5b6f8daf3b WatchSource:0}: Error finding container 79352a93e604a566c5dbb7bf9b674c64a2bd656beca9927e5f826c5b6f8daf3b: Status 404 returned error can't find the container with id 79352a93e604a566c5dbb7bf9b674c64a2bd656beca9927e5f826c5b6f8daf3b Jul 15 22:34:00.284902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:00.284868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:00.285116 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:00.284927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:00.285116 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:00.285004 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 22:34:00.285116 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:00.285036 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 22:34:00.285116 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:00.285076 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:04.285060042 +0000 UTC m=+55.225262570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : configmap "kube-rbac-proxy" not found Jul 15 22:34:00.285116 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:00.285091 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:04.285084701 +0000 UTC m=+55.225287228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : secret "insights-runtime-extractor-tls" not found Jul 15 22:34:00.817990 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:00.817955 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" event={"ID":"9d8e7823-5966-41cb-8a72-79ba54b7f74d","Type":"ContainerStarted","Data":"79352a93e604a566c5dbb7bf9b674c64a2bd656beca9927e5f826c5b6f8daf3b"} Jul 15 22:34:01.007698 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:01.007660 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:02.199990 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:02.199946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:34:02.200498 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:02.199998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:34:02.200498 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:02.200109 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 22:34:02.200498 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:02.200191 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls podName:bd6596a2-25e0-4482-bd65-38fb27ddfe8d nodeName:}" failed. No retries permitted until 2026-07-15 22:34:18.200154975 +0000 UTC m=+69.140357503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls") pod "dns-default-rsk5w" (UID: "bd6596a2-25e0-4482-bd65-38fb27ddfe8d") : secret "dns-default-metrics-tls" not found Jul 15 22:34:02.200498 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:02.200233 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 22:34:02.200498 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:02.200252 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc5875b8d-qlg76: secret "image-registry-tls" not found Jul 15 22:34:02.200498 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:02.200298 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls podName:28a47249-51b6-40ed-8050-0715614d7a82 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:18.200286102 +0000 UTC m=+69.140488629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls") pod "image-registry-fc5875b8d-qlg76" (UID: "28a47249-51b6-40ed-8050-0715614d7a82") : secret "image-registry-tls" not found Jul 15 22:34:02.825373 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:02.825288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" event={"ID":"9d8e7823-5966-41cb-8a72-79ba54b7f74d","Type":"ContainerStarted","Data":"775c8348a1c2acf623791f730c3e1c29826e2ebccb48d24b8ee8b06355046f12"} Jul 15 22:34:02.842805 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:02.842748 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-w8wx9" podStartSLOduration=1.5390570399999999 podStartE2EDuration="3.842731456s" totalImagesPullingTime="2.303674416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:02.841951023 +0000 UTC m=+53.782153571" watchObservedRunningTime="2026-07-15 22:34:02.842731456 +0000 UTC m=+53.782934029" Jul 15 22:34:04.198009 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:04.197971 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:04.317042 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:04.317007 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:04.318497 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:04.318472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:04.318617 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:04.318601 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:04.318688 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:04.318646 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 22:34:04.318742 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:04.318702 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 22:34:04.318806 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:04.318753 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:12.31871529 +0000 UTC m=+63.258917835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : secret "insights-runtime-extractor-tls" not found Jul 15 22:34:04.318806 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:04.318775 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:12.318764021 +0000 UTC m=+63.258966551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : configmap "kube-rbac-proxy" not found Jul 15 22:34:04.420003 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:04.419949 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:04.520694 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:04.520600 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:04.620922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:04.620887 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:05.071937 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:05.071904 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:05.282810 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:05.282769 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:06.471198 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:06.470609 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:06.664612 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:06.664569 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:06.877667 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:06.877641 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:06.929297 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:06.929264 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:07.059491 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:07.059456 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:07.262198 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:07.262094 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:07.790988 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:07.790950 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:07.891044 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:07.891009 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:07.993918 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:07.993888 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:08.091647 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:08.091570 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:08.266627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:08.266589 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:08.461682 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:08.461628 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:08.516245 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:08.516216 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:08.859840 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:08.859747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:09.068406 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:09.068375 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:09.464726 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:09.464693 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:10.261974 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:10.261934 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:10.365121 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:10.365090 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:10.664337 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:10.664297 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:10.760235 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:10.760209 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qj5gc" Jul 15 22:34:10.862444 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:10.862414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:11.071546 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:11.071448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:11.265230 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:11.265196 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:11.722326 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:11.722292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:11.827402 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:11.827361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:12.273799 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:12.273760 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:12.388189 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:12.388112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:12.388388 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:12.388214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:12.388388 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:12.388335 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 22:34:12.388508 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:12.388415 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls podName:9bfad52c-9805-44c2-b8e0-5ae8243987d0 nodeName:}" failed. No retries permitted until 2026-07-15 22:34:28.388396499 +0000 UTC m=+79.328599046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4tsd" (UID: "9bfad52c-9805-44c2-b8e0-5ae8243987d0") : secret "insights-runtime-extractor-tls" not found Jul 15 22:34:12.388666 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:12.388642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9bfad52c-9805-44c2-b8e0-5ae8243987d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:12.470577 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:12.470548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:12.679808 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:12.679775 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:13.073104 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:13.073028 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:13.864306 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:13.864266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:14.063711 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:14.063673 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:14.460363 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:14.460329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:14.664617 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:14.664582 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:14.858618 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:14.858538 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:15.413575 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.413534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:34:15.413932 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.413612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:34:15.416408 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.416388 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 22:34:15.416465 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.416392 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 22:34:15.426836 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.426808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a63fe02-5823-4b44-b017-059e1e8ab33d-original-pull-secret\") pod \"global-pull-secret-syncer-gp62w\" (UID: \"2a63fe02-5823-4b44-b017-059e1e8ab33d\") " pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:34:15.426836 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.426827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9b6737-f701-4ecb-86f1-0c25f73efdf6-metrics-certs\") pod \"network-metrics-daemon-jhmzm\" (UID: \"7b9b6737-f701-4ecb-86f1-0c25f73efdf6\") " pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:34:15.514632 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.514591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:34:15.517600 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.517582 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 22:34:15.528154 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.528132 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 22:34:15.538931 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.538910 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbkq\" (UniqueName: \"kubernetes.io/projected/a3257827-435d-40d8-ad19-955d7041302e-kube-api-access-gmbkq\") pod \"network-check-target-v26ns\" (UID: \"a3257827-435d-40d8-ad19-955d7041302e\") " pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:34:15.701024 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.700937 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-84k7s\"" Jul 15 22:34:15.703205 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.703188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gp62w" Jul 15 22:34:15.708347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.708324 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:34:15.710766 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.710751 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmq28\"" Jul 15 22:34:15.719122 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.719100 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhmzm" Jul 15 22:34:15.853029 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.852933 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gp62w"] Jul 15 22:34:15.856859 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:15.856829 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a63fe02_5823_4b44_b017_059e1e8ab33d.slice/crio-ece0f9638df3ae20120a029ff22a7a019c3c11da79263181fb71776aaf082ce6 WatchSource:0}: Error finding container ece0f9638df3ae20120a029ff22a7a019c3c11da79263181fb71776aaf082ce6: Status 404 returned error can't find the container with id ece0f9638df3ae20120a029ff22a7a019c3c11da79263181fb71776aaf082ce6 Jul 15 22:34:15.872527 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.872504 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v26ns"] Jul 15 22:34:15.874761 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:15.874731 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3257827_435d_40d8_ad19_955d7041302e.slice/crio-4e896b272dc9e43533495229d233c931fc540172cdf60165d5ac12791ad38cd9 WatchSource:0}: Error finding container 4e896b272dc9e43533495229d233c931fc540172cdf60165d5ac12791ad38cd9: Status 404 returned error can't find the container with id 4e896b272dc9e43533495229d233c931fc540172cdf60165d5ac12791ad38cd9 Jul 15 22:34:15.886970 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:15.886946 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhmzm"] Jul 15 22:34:15.889764 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:15.889744 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9b6737_f701_4ecb_86f1_0c25f73efdf6.slice/crio-dc62bd4ce5548f7fa6f42e7738d2287a897371fbcccd9c72ad765aadee7067f5 WatchSource:0}: Error finding container dc62bd4ce5548f7fa6f42e7738d2287a897371fbcccd9c72ad765aadee7067f5: Status 404 returned error can't find the container with id dc62bd4ce5548f7fa6f42e7738d2287a897371fbcccd9c72ad765aadee7067f5 Jul 15 22:34:16.866891 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:16.866853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhmzm" event={"ID":"7b9b6737-f701-4ecb-86f1-0c25f73efdf6","Type":"ContainerStarted","Data":"dc62bd4ce5548f7fa6f42e7738d2287a897371fbcccd9c72ad765aadee7067f5"} Jul 15 22:34:16.868765 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:16.868733 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gp62w" event={"ID":"2a63fe02-5823-4b44-b017-059e1e8ab33d","Type":"ContainerStarted","Data":"ece0f9638df3ae20120a029ff22a7a019c3c11da79263181fb71776aaf082ce6"} Jul 15 22:34:16.870861 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:16.870825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v26ns" event={"ID":"a3257827-435d-40d8-ad19-955d7041302e","Type":"ContainerStarted","Data":"4e896b272dc9e43533495229d233c931fc540172cdf60165d5ac12791ad38cd9"} Jul 15 22:34:17.877314 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:17.877277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhmzm" event={"ID":"7b9b6737-f701-4ecb-86f1-0c25f73efdf6","Type":"ContainerStarted","Data":"e02cfe6604d8c67f6007a637e5dea3ffe6225a8aa3ef26e53331b010aafde9c8"} Jul 15 22:34:17.877809 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:17.877338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhmzm" event={"ID":"7b9b6737-f701-4ecb-86f1-0c25f73efdf6","Type":"ContainerStarted","Data":"1b46efa08749b0307e8adb6f26286d53d782e6dc43636e4946c2ec2b6281362c"} Jul 15 22:34:17.895712 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:17.895643 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jhmzm" podStartSLOduration=66.558081059 podStartE2EDuration="1m7.895620405s" totalImagesPullingTime="1.337539346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:17.895282783 +0000 UTC m=+68.835485332" watchObservedRunningTime="2026-07-15 22:34:17.895620405 +0000 UTC m=+68.835822965" Jul 15 22:34:18.237914 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:18.237877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:34:18.238111 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:18.237931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:34:18.241016 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:18.240989 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6596a2-25e0-4482-bd65-38fb27ddfe8d-metrics-tls\") pod \"dns-default-rsk5w\" (UID: \"bd6596a2-25e0-4482-bd65-38fb27ddfe8d\") " pod="openshift-dns/dns-default-rsk5w" Jul 15 22:34:18.241132 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:18.241072 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"image-registry-fc5875b8d-qlg76\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:34:18.475819 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:18.475786 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7pd8h\"" Jul 15 22:34:18.482084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:18.482055 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-trnq4\"" Jul 15 22:34:18.483152 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:18.483127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:34:18.489874 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:18.489847 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rsk5w" Jul 15 22:34:21.054801 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.054566 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fc5875b8d-qlg76"] Jul 15 22:34:21.058425 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:21.058397 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a47249_51b6_40ed_8050_0715614d7a82.slice/crio-0d67c16799eb34e4f899fe9e11a962e345faf5de837d16ccf27bb18a1873cff6 WatchSource:0}: Error finding container 0d67c16799eb34e4f899fe9e11a962e345faf5de837d16ccf27bb18a1873cff6: Status 404 returned error can't find the container with id 0d67c16799eb34e4f899fe9e11a962e345faf5de837d16ccf27bb18a1873cff6 Jul 15 22:34:21.072144 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.072087 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rsk5w"] Jul 15 22:34:21.076605 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:21.076576 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6596a2_25e0_4482_bd65_38fb27ddfe8d.slice/crio-18e4eea39a0707dc7369e194f0a0ea5bbdf8efedf6a7e075fa15eaebdd3ace07 WatchSource:0}: Error finding container 18e4eea39a0707dc7369e194f0a0ea5bbdf8efedf6a7e075fa15eaebdd3ace07: Status 404 returned error can't find the container with id 18e4eea39a0707dc7369e194f0a0ea5bbdf8efedf6a7e075fa15eaebdd3ace07 Jul 15 22:34:21.894031 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.893994 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gp62w" event={"ID":"2a63fe02-5823-4b44-b017-059e1e8ab33d","Type":"ContainerStarted","Data":"dc341d18f66c500c6ccbbf76b0dc579e98619cd461b4d2fda1002a7f105a02c1"} Jul 15 22:34:21.895481 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.895454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" event={"ID":"28a47249-51b6-40ed-8050-0715614d7a82","Type":"ContainerStarted","Data":"1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b"} Jul 15 22:34:21.895481 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.895504 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:34:21.895481 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.895523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" event={"ID":"28a47249-51b6-40ed-8050-0715614d7a82","Type":"ContainerStarted","Data":"0d67c16799eb34e4f899fe9e11a962e345faf5de837d16ccf27bb18a1873cff6"} Jul 15 22:34:21.896903 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.896866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v26ns" event={"ID":"a3257827-435d-40d8-ad19-955d7041302e","Type":"ContainerStarted","Data":"89b8648ca4d06a44a4a47e3dd948cc7d90c10e8dba4c3cd92b67b327720bf5f5"} Jul 15 22:34:21.897025 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.897011 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:34:21.897933 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.897914 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rsk5w" event={"ID":"bd6596a2-25e0-4482-bd65-38fb27ddfe8d","Type":"ContainerStarted","Data":"18e4eea39a0707dc7369e194f0a0ea5bbdf8efedf6a7e075fa15eaebdd3ace07"} Jul 15 22:34:21.908975 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.908930 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gp62w" podStartSLOduration=66.843334194 podStartE2EDuration="1m11.908915958s" totalImagesPullingTime="5.065581764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:21.908669266 +0000 UTC m=+72.848871818" watchObservedRunningTime="2026-07-15 22:34:21.908915958 +0000 UTC m=+72.849118508" Jul 15 22:34:21.926171 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.926131 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v26ns" podStartSLOduration=66.888741876 podStartE2EDuration="1m11.926118189s" totalImagesPullingTime="5.037376313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:21.925371096 +0000 UTC m=+72.865573646" watchObservedRunningTime="2026-07-15 22:34:21.926118189 +0000 UTC m=+72.866320739" Jul 15 22:34:21.947895 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:21.947850 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" podStartSLOduration=48.947835429 podStartE2EDuration="48.947835429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:21.946913507 +0000 UTC m=+72.887116057" watchObservedRunningTime="2026-07-15 22:34:21.947835429 +0000 UTC m=+72.888038110" Jul 15 22:34:22.040334 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.040302 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb"] Jul 15 22:34:22.043871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.043851 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv"] Jul 15 22:34:22.043989 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.043970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.046519 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.046496 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.046797 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.046775 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 22:34:22.046986 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.046969 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 22:34:22.047068 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.047022 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 22:34:22.047129 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.047089 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 22:34:22.050264 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.050243 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 22:34:22.050397 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.050316 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 22:34:22.050781 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.050763 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 22:34:22.050867 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.050785 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 22:34:22.057850 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.057827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb"] Jul 15 22:34:22.072721 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.072695 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv"] Jul 15 22:34:22.106803 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.106768 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fc5875b8d-qlg76"] Jul 15 22:34:22.162609 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.162528 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-845987996-slz86"] Jul 15 22:34:22.166354 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.166336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.169941 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.169916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d814143-f603-4ad3-a38a-611e38191235-tmp\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.170062 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.169951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.170062 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.169977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1d814143-f603-4ad3-a38a-611e38191235-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.170062 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.170046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.170203 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.170070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vd96\" (UniqueName: \"kubernetes.io/projected/1d814143-f603-4ad3-a38a-611e38191235-kube-api-access-2vd96\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.170203 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.170146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/687e00bf-eacb-4967-9b3e-49bc71a032ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.170292 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.170206 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-hub\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.170292 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.170237 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-ca\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.170292 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.170260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twvzq\" (UniqueName: \"kubernetes.io/projected/687e00bf-eacb-4967-9b3e-49bc71a032ba-kube-api-access-twvzq\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.181946 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.181917 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:22.183096 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.183042 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-845987996-slz86"] Jul 15 22:34:22.271071 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271026 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7743a76-2485-4a98-9c05-945ce06ca4d1-registry-certificates\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.271259 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.271259 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vd96\" (UniqueName: \"kubernetes.io/projected/1d814143-f603-4ad3-a38a-611e38191235-kube-api-access-2vd96\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.271259 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-bound-sa-token\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.271259 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/687e00bf-eacb-4967-9b3e-49bc71a032ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.271259 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271207 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-hub\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.271259 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhh62\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-kube-api-access-zhh62\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.271259 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7743a76-2485-4a98-9c05-945ce06ca4d1-image-registry-private-configuration\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271270 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7743a76-2485-4a98-9c05-945ce06ca4d1-trusted-ca\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-ca\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twvzq\" (UniqueName: \"kubernetes.io/projected/687e00bf-eacb-4967-9b3e-49bc71a032ba-kube-api-access-twvzq\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7743a76-2485-4a98-9c05-945ce06ca4d1-installation-pull-secrets\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d814143-f603-4ad3-a38a-611e38191235-tmp\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7743a76-2485-4a98-9c05-945ce06ca4d1-ca-trust-extracted\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1d814143-f603-4ad3-a38a-611e38191235-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.271627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271509 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-registry-tls\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.272058 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.271940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d814143-f603-4ad3-a38a-611e38191235-tmp\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.272058 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.272018 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/687e00bf-eacb-4967-9b3e-49bc71a032ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.274494 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.274468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-ca\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.274632 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.274568 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.274705 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.274648 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.275105 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.275076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1d814143-f603-4ad3-a38a-611e38191235-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.275105 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.275093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/687e00bf-eacb-4967-9b3e-49bc71a032ba-hub\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.286768 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.286746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vd96\" (UniqueName: \"kubernetes.io/projected/1d814143-f603-4ad3-a38a-611e38191235-kube-api-access-2vd96\") pod \"klusterlet-addon-workmgr-5c787646c7-jl2vb\" (UID: \"1d814143-f603-4ad3-a38a-611e38191235\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.286891 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.286848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twvzq\" (UniqueName: \"kubernetes.io/projected/687e00bf-eacb-4967-9b3e-49bc71a032ba-kube-api-access-twvzq\") pod \"cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv\" (UID: \"687e00bf-eacb-4967-9b3e-49bc71a032ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.305526 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.305498 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:22.355060 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.355026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:22.370777 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.370750 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" Jul 15 22:34:22.371978 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.371955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7743a76-2485-4a98-9c05-945ce06ca4d1-ca-trust-extracted\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372110 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.371994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-registry-tls\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372110 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.372022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7743a76-2485-4a98-9c05-945ce06ca4d1-registry-certificates\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372110 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.372057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-bound-sa-token\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372110 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.372096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhh62\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-kube-api-access-zhh62\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.372123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7743a76-2485-4a98-9c05-945ce06ca4d1-image-registry-private-configuration\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.372147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7743a76-2485-4a98-9c05-945ce06ca4d1-trusted-ca\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.372204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7743a76-2485-4a98-9c05-945ce06ca4d1-installation-pull-secrets\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372500 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.372375 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7743a76-2485-4a98-9c05-945ce06ca4d1-ca-trust-extracted\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.372880 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.372860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7743a76-2485-4a98-9c05-945ce06ca4d1-registry-certificates\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.373945 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.373879 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7743a76-2485-4a98-9c05-945ce06ca4d1-trusted-ca\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.374853 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.374809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7743a76-2485-4a98-9c05-945ce06ca4d1-installation-pull-secrets\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.375245 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.375207 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-registry-tls\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.375328 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.375308 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7743a76-2485-4a98-9c05-945ce06ca4d1-image-registry-private-configuration\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.382955 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.382929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-bound-sa-token\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.383929 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.383911 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhh62\" (UniqueName: \"kubernetes.io/projected/f7743a76-2485-4a98-9c05-945ce06ca4d1-kube-api-access-zhh62\") pod \"image-registry-845987996-slz86\" (UID: \"f7743a76-2485-4a98-9c05-945ce06ca4d1\") " pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.415953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.415832 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:22.476310 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.476282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.518541 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.518505 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb"] Jul 15 22:34:22.522802 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:22.522774 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d814143_f603_4ad3_a38a_611e38191235.slice/crio-f5f27c27316eafddfde94c5f3186b28a504871e1ee556ace1bf5b7c8b037d93d WatchSource:0}: Error finding container f5f27c27316eafddfde94c5f3186b28a504871e1ee556ace1bf5b7c8b037d93d: Status 404 returned error can't find the container with id f5f27c27316eafddfde94c5f3186b28a504871e1ee556ace1bf5b7c8b037d93d Jul 15 22:34:22.536819 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.536770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv"] Jul 15 22:34:22.632334 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.632303 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-845987996-slz86"] Jul 15 22:34:22.635008 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:22.634973 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7743a76_2485_4a98_9c05_945ce06ca4d1.slice/crio-f99c20222174c7ec3535b7a4cd6ae513fb64eaa40092d58ac56e796780b9679b WatchSource:0}: Error finding container f99c20222174c7ec3535b7a4cd6ae513fb64eaa40092d58ac56e796780b9679b: Status 404 returned error can't find the container with id f99c20222174c7ec3535b7a4cd6ae513fb64eaa40092d58ac56e796780b9679b Jul 15 22:34:22.905545 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.905504 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-845987996-slz86" event={"ID":"f7743a76-2485-4a98-9c05-945ce06ca4d1","Type":"ContainerStarted","Data":"0ef3561d77982d25543b0f5ea93c003ad0c30026480553fec2a730e731ec3f24"} Jul 15 22:34:22.905722 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.905569 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-845987996-slz86" event={"ID":"f7743a76-2485-4a98-9c05-945ce06ca4d1","Type":"ContainerStarted","Data":"f99c20222174c7ec3535b7a4cd6ae513fb64eaa40092d58ac56e796780b9679b"} Jul 15 22:34:22.905722 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.905596 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:22.907260 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.907206 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" event={"ID":"687e00bf-eacb-4967-9b3e-49bc71a032ba","Type":"ContainerStarted","Data":"bf9a6559c175354ea0b9d8ba7a4baab37b4253a209b28f8ec802375d5d496969"} Jul 15 22:34:22.908952 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.908926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" event={"ID":"1d814143-f603-4ad3-a38a-611e38191235","Type":"ContainerStarted","Data":"f5f27c27316eafddfde94c5f3186b28a504871e1ee556ace1bf5b7c8b037d93d"} Jul 15 22:34:22.925364 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:22.925222 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-845987996-slz86" podStartSLOduration=0.925200888 podStartE2EDuration="925.200888ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:22.924333934 +0000 UTC m=+73.864536508" watchObservedRunningTime="2026-07-15 22:34:22.925200888 +0000 UTC m=+73.865403440" Jul 15 22:34:23.494836 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:23.494766 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:23.589598 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:23.589506 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:23.871780 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:23.871668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:23.918621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:23.918584 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rsk5w" event={"ID":"bd6596a2-25e0-4482-bd65-38fb27ddfe8d","Type":"ContainerStarted","Data":"98a5e8c9b73bba31e0b3c33102b93bf0cd588160b853e83dd6f5ad0032677173"} Jul 15 22:34:23.918824 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:23.918652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rsk5w" event={"ID":"bd6596a2-25e0-4482-bd65-38fb27ddfe8d","Type":"ContainerStarted","Data":"dbf308070d56fc5afb2cc2198c931bb4592e49418668ba3f6395b0bb767c10af"} Jul 15 22:34:23.918824 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:23.918683 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rsk5w" Jul 15 22:34:23.954628 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:23.954549 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:23.965465 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:23.965408 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rsk5w" podStartSLOduration=35.688374528 podStartE2EDuration="37.965385005s" totalImagesPullingTime="2.277010477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:23.965074146 +0000 UTC m=+74.905276697" watchObservedRunningTime="2026-07-15 22:34:23.965385005 +0000 UTC m=+74.905587556" Jul 15 22:34:24.037338 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:24.037268 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:24.095143 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:24.095057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:24.327709 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:24.327589 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:24.459072 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:24.458978 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:24.558634 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:24.558523 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:24.654945 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:24.654825 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:24.761603 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:24.761544 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:25.014487 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:25.014345 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:25.088854 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:25.088743 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:25.180057 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:25.179957 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:25.246785 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:25.246747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:25.312480 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:25.312383 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:25.413623 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:25.413586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:25.489435 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:25.489376 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:26.292299 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:26.292256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:26.352232 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:26.352203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:27.409661 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:27.409625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:27.476299 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:27.476266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:28.430160 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.430122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:28.432725 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.432697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9bfad52c-9805-44c2-b8e0-5ae8243987d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4tsd\" (UID: \"9bfad52c-9805-44c2-b8e0-5ae8243987d0\") " pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:28.436077 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.436058 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkr96\"" Jul 15 22:34:28.444368 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.444349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4tsd" Jul 15 22:34:28.589857 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.589834 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4tsd"] Jul 15 22:34:28.592094 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:28.592070 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfad52c_9805_44c2_b8e0_5ae8243987d0.slice/crio-0e244a94b0ea789af625a3271ed891b51e954ed9606c9bb8e6f17830d947d285 WatchSource:0}: Error finding container 0e244a94b0ea789af625a3271ed891b51e954ed9606c9bb8e6f17830d947d285: Status 404 returned error can't find the container with id 0e244a94b0ea789af625a3271ed891b51e954ed9606c9bb8e6f17830d947d285 Jul 15 22:34:28.939302 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.939260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" event={"ID":"687e00bf-eacb-4967-9b3e-49bc71a032ba","Type":"ContainerStarted","Data":"5ef4cb3b77a5420a77bc36a1ec92b0fbde98167e974690b4c315c73b6491101b"} Jul 15 22:34:28.939516 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.939315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" event={"ID":"687e00bf-eacb-4967-9b3e-49bc71a032ba","Type":"ContainerStarted","Data":"f2597c89b1a125fe7001ed5cd2a2cfeae65a4ff2bfbdcfe4ffaa7d6f625ce6ee"} Jul 15 22:34:28.939516 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.939338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" event={"ID":"687e00bf-eacb-4967-9b3e-49bc71a032ba","Type":"ContainerStarted","Data":"ac43a57f99883b5c7180ce9138aaf13670df56c3581fa6192ba121c412fde1e9"} Jul 15 22:34:28.940522 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.940496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4tsd" event={"ID":"9bfad52c-9805-44c2-b8e0-5ae8243987d0","Type":"ContainerStarted","Data":"0e244a94b0ea789af625a3271ed891b51e954ed9606c9bb8e6f17830d947d285"} Jul 15 22:34:28.941907 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.941888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" event={"ID":"1d814143-f603-4ad3-a38a-611e38191235","Type":"ContainerStarted","Data":"7a4f16bbac3d5070f5a801010dc903951cae80fe93d528a12e2cab69736f1f33"} Jul 15 22:34:28.942099 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.942081 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:28.943895 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.943868 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" Jul 15 22:34:28.959856 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.959809 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fbc6cbfc5-kztwv" podStartSLOduration=1.522858572 podStartE2EDuration="6.959794707s" totalImagesPullingTime="5.436936135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:28.958446634 +0000 UTC m=+79.898649198" watchObservedRunningTime="2026-07-15 22:34:28.959794707 +0000 UTC m=+79.899997256" Jul 15 22:34:28.975499 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:28.975451 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c787646c7-jl2vb" podStartSLOduration=1.516497544 podStartE2EDuration="6.97543892s" totalImagesPullingTime="5.458941376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:28.975093059 +0000 UTC m=+79.915295621" watchObservedRunningTime="2026-07-15 22:34:28.97543892 +0000 UTC m=+79.915641469" Jul 15 22:34:29.846704 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:29.846153 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:29.913147 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:29.913105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:32.026678 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:32.026640 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:32.110530 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:32.110493 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:32.387765 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:32.387708 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:33.639881 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:33.639845 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:33.734661 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:33.734605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:33.926243 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:33.926149 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rsk5w" Jul 15 22:34:34.951272 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:34.951239 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:34.971964 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:34.971931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4tsd" event={"ID":"9bfad52c-9805-44c2-b8e0-5ae8243987d0","Type":"ContainerStarted","Data":"0c3c985c3b6f72e0923811cb0bd163b7caad44453f59e0c5c4c9874dfc8be3a4"} Jul 15 22:34:35.037793 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:35.037761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:35.546310 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:35.546261 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:35.664915 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:35.664858 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:36.025983 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:36.025940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:36.086017 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:36.085963 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:38.665211 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:38.665178 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:38.735993 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:38.735951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:38.988203 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:38.988099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4tsd" event={"ID":"9bfad52c-9805-44c2-b8e0-5ae8243987d0","Type":"ContainerStarted","Data":"28c2fb674a1e342f81095c0f2d97c9e0305a06740ae99cb61f645e76cba060dc"} Jul 15 22:34:39.483829 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:39.483790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:39.548679 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:39.548637 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:40.018813 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:40.018777 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:40.090947 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:40.090899 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:42.481186 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:42.481131 2565 patch_prober.go:28] interesting pod/image-registry-845987996-slz86 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 15 22:34:42.481572 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:42.481204 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-845987996-slz86" podUID="f7743a76-2485-4a98-9c05-945ce06ca4d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 22:34:43.005182 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:43.005135 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4tsd" event={"ID":"9bfad52c-9805-44c2-b8e0-5ae8243987d0","Type":"ContainerStarted","Data":"fb86f03e9004f0a88a6b1cda37d280dec0f77e17c3bbc91679821f59563eabc9"} Jul 15 22:34:43.037152 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:43.037096 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r4tsd" podStartSLOduration=33.799423138 podStartE2EDuration="47.037079439s" totalImagesPullingTime="13.237656301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:33:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:43.036586652 +0000 UTC m=+93.976789201" watchObservedRunningTime="2026-07-15 22:34:43.037079439 +0000 UTC m=+93.977281988" Jul 15 22:34:43.142218 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:43.142153 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:43.233022 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:43.232988 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:43.923823 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:43.923797 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-845987996-slz86" Jul 15 22:34:43.924663 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:43.924646 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:34:44.826189 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:44.826135 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:44.939428 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:44.939388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:46.945068 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:46.945031 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:47.062428 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:47.062387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:48.964321 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:48.964274 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" podUID="28a47249-51b6-40ed-8050-0715614d7a82" containerName="registry" containerID="cri-o://1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b" gracePeriod=30 Jul 15 22:34:49.214915 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.214881 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:49.231724 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.231701 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:34:49.296362 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296326 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-trusted-ca\") pod \"28a47249-51b6-40ed-8050-0715614d7a82\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " Jul 15 22:34:49.296521 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296377 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28a47249-51b6-40ed-8050-0715614d7a82-ca-trust-extracted\") pod \"28a47249-51b6-40ed-8050-0715614d7a82\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " Jul 15 22:34:49.296521 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296401 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-bound-sa-token\") pod \"28a47249-51b6-40ed-8050-0715614d7a82\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " Jul 15 22:34:49.296521 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296425 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") pod \"28a47249-51b6-40ed-8050-0715614d7a82\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " Jul 15 22:34:49.296639 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296605 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-installation-pull-secrets\") pod \"28a47249-51b6-40ed-8050-0715614d7a82\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " Jul 15 22:34:49.296690 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296642 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-registry-certificates\") pod \"28a47249-51b6-40ed-8050-0715614d7a82\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " Jul 15 22:34:49.296742 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296685 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-image-registry-private-configuration\") pod \"28a47249-51b6-40ed-8050-0715614d7a82\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " Jul 15 22:34:49.296742 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296712 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4jhd\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-kube-api-access-h4jhd\") pod \"28a47249-51b6-40ed-8050-0715614d7a82\" (UID: \"28a47249-51b6-40ed-8050-0715614d7a82\") " Jul 15 22:34:49.296841 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.296778 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "28a47249-51b6-40ed-8050-0715614d7a82" (UID: "28a47249-51b6-40ed-8050-0715614d7a82"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 22:34:49.297122 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.297085 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-trusted-ca\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:34:49.297364 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.297326 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "28a47249-51b6-40ed-8050-0715614d7a82" (UID: "28a47249-51b6-40ed-8050-0715614d7a82"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 22:34:49.299070 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.299025 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "28a47249-51b6-40ed-8050-0715614d7a82" (UID: "28a47249-51b6-40ed-8050-0715614d7a82"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:34:49.299292 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.299272 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "28a47249-51b6-40ed-8050-0715614d7a82" (UID: "28a47249-51b6-40ed-8050-0715614d7a82"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:34:49.299369 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.299303 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "28a47249-51b6-40ed-8050-0715614d7a82" (UID: "28a47249-51b6-40ed-8050-0715614d7a82"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:34:49.299525 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.299507 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-kube-api-access-h4jhd" (OuterVolumeSpecName: "kube-api-access-h4jhd") pod "28a47249-51b6-40ed-8050-0715614d7a82" (UID: "28a47249-51b6-40ed-8050-0715614d7a82"). InnerVolumeSpecName "kube-api-access-h4jhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:34:49.299577 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.299523 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "28a47249-51b6-40ed-8050-0715614d7a82" (UID: "28a47249-51b6-40ed-8050-0715614d7a82"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:34:49.305475 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.305453 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a47249-51b6-40ed-8050-0715614d7a82-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "28a47249-51b6-40ed-8050-0715614d7a82" (UID: "28a47249-51b6-40ed-8050-0715614d7a82"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:34:49.319570 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.319543 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:49.398068 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.398027 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28a47249-51b6-40ed-8050-0715614d7a82-ca-trust-extracted\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:34:49.398068 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.398065 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-bound-sa-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:34:49.398314 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.398079 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-registry-tls\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:34:49.398314 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.398090 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-installation-pull-secrets\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:34:49.398314 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.398104 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28a47249-51b6-40ed-8050-0715614d7a82-registry-certificates\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:34:49.398314 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.398117 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28a47249-51b6-40ed-8050-0715614d7a82-image-registry-private-configuration\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:34:49.398314 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.398129 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4jhd\" (UniqueName: \"kubernetes.io/projected/28a47249-51b6-40ed-8050-0715614d7a82-kube-api-access-h4jhd\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:34:49.852774 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:49.852689 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:50.034212 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.034149 2565 generic.go:356] "Generic (PLEG): container finished" podID="28a47249-51b6-40ed-8050-0715614d7a82" containerID="1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b" exitCode=0 Jul 15 22:34:50.034635 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.034209 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" event={"ID":"28a47249-51b6-40ed-8050-0715614d7a82","Type":"ContainerDied","Data":"1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b"} Jul 15 22:34:50.034635 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.034227 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" Jul 15 22:34:50.034635 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.034269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc5875b8d-qlg76" event={"ID":"28a47249-51b6-40ed-8050-0715614d7a82","Type":"ContainerDied","Data":"0d67c16799eb34e4f899fe9e11a962e345faf5de837d16ccf27bb18a1873cff6"} Jul 15 22:34:50.034635 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.034291 2565 scope.go:117] "RemoveContainer" containerID="1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b" Jul 15 22:34:50.042394 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.042377 2565 scope.go:117] "RemoveContainer" containerID="1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b" Jul 15 22:34:50.042700 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:34:50.042682 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b\": container with ID starting with 1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b not found: ID does not exist" containerID="1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b" Jul 15 22:34:50.042744 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.042709 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b"} err="failed to get container status \"1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b\": rpc error: code = NotFound desc = could not find container \"1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b\": container with ID starting with 1802ad31ae8af255d21c09933403fe2c485c074a50091c183dae25c3c26b893b not found: ID does not exist" Jul 15 22:34:50.074111 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.074076 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fc5875b8d-qlg76"] Jul 15 22:34:50.090652 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:50.090625 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fc5875b8d-qlg76"] Jul 15 22:34:51.583513 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:51.583478 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28a47249-51b6-40ed-8050-0715614d7a82" path="/var/lib/kubelet/pods/28a47249-51b6-40ed-8050-0715614d7a82/volumes" Jul 15 22:34:52.282460 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:52.282425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:52.457590 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:52.457544 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:52.694133 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:52.694100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:52.901953 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:52.901910 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:52.911778 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:52.911749 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v26ns" Jul 15 22:34:53.083526 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.083445 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zc8w9"] Jul 15 22:34:53.083684 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.083672 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a47249-51b6-40ed-8050-0715614d7a82" containerName="registry" Jul 15 22:34:53.083724 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.083686 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a47249-51b6-40ed-8050-0715614d7a82" containerName="registry" Jul 15 22:34:53.083757 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.083737 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a47249-51b6-40ed-8050-0715614d7a82" containerName="registry" Jul 15 22:34:53.087580 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.087559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.092908 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.092885 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 22:34:53.093047 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.092947 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 22:34:53.093188 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.093143 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 22:34:53.093436 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.093412 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 22:34:53.096265 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.096244 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 22:34:53.096375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.096308 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j6k2f\"" Jul 15 22:34:53.097056 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.097041 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 22:34:53.112341 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.112312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:53.124547 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124525 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-root\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.124655 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-tls\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.124655 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.124655 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4mf\" (UniqueName: \"kubernetes.io/projected/c668433e-ff6c-4321-82e4-a26430446e1b-kube-api-access-zq4mf\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.124814 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-textfile\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.124814 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-wtmp\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.124814 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.124951 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124829 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-sys\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.124951 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.124857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c668433e-ff6c-4321-82e4-a26430446e1b-metrics-client-ca\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225231 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-sys\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225231 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c668433e-ff6c-4321-82e4-a26430446e1b-metrics-client-ca\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-root\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-tls\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-root\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225331 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-sys\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4mf\" (UniqueName: \"kubernetes.io/projected/c668433e-ff6c-4321-82e4-a26430446e1b-kube-api-access-zq4mf\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-textfile\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-wtmp\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225495 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225931 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-wtmp\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.225931 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-textfile\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.226019 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.225940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c668433e-ff6c-4321-82e4-a26430446e1b-metrics-client-ca\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.226111 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.226088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.227851 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.227829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.228026 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.228009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c668433e-ff6c-4321-82e4-a26430446e1b-node-exporter-tls\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.250939 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.250896 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4mf\" (UniqueName: \"kubernetes.io/projected/c668433e-ff6c-4321-82e4-a26430446e1b-kube-api-access-zq4mf\") pod \"node-exporter-zc8w9\" (UID: \"c668433e-ff6c-4321-82e4-a26430446e1b\") " pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.398241 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.398207 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zc8w9" Jul 15 22:34:53.398748 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.398725 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:53.408974 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:34:53.408946 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc668433e_ff6c_4321_82e4_a26430446e1b.slice/crio-9c91d1edd9270dc7dbea632551cc987749cfd867f7b33314a956094691e475ff WatchSource:0}: Error finding container 9c91d1edd9270dc7dbea632551cc987749cfd867f7b33314a956094691e475ff: Status 404 returned error can't find the container with id 9c91d1edd9270dc7dbea632551cc987749cfd867f7b33314a956094691e475ff Jul 15 22:34:53.589846 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.589809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:53.860404 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:53.860311 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:54.040920 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:54.040878 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:54.049138 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:54.049062 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zc8w9" event={"ID":"c668433e-ff6c-4321-82e4-a26430446e1b","Type":"ContainerStarted","Data":"9c91d1edd9270dc7dbea632551cc987749cfd867f7b33314a956094691e475ff"} Jul 15 22:34:54.240088 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:54.240049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:55.053974 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:55.053937 2565 generic.go:356] "Generic (PLEG): container finished" podID="c668433e-ff6c-4321-82e4-a26430446e1b" containerID="d987eda07cc4dfff169fa0a4b135cdf5555a4fc1cd7d259d17e5bf246dc49b40" exitCode=0 Jul 15 22:34:55.054379 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:55.053992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zc8w9" event={"ID":"c668433e-ff6c-4321-82e4-a26430446e1b","Type":"ContainerDied","Data":"d987eda07cc4dfff169fa0a4b135cdf5555a4fc1cd7d259d17e5bf246dc49b40"} Jul 15 22:34:55.444539 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:55.444505 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:55.567686 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:55.567651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:56.038598 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:56.038562 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:56.060566 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:56.060522 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zc8w9" event={"ID":"c668433e-ff6c-4321-82e4-a26430446e1b","Type":"ContainerStarted","Data":"4011d2bfe34633fe2759c823dfdecbca728d2d545d41aae6719e274a2b1b62fb"} Jul 15 22:34:56.061040 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:56.060590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zc8w9" event={"ID":"c668433e-ff6c-4321-82e4-a26430446e1b","Type":"ContainerStarted","Data":"b0f8b6a22535325f1337f22139f5cd72283e14a65acba9e0a695f58d10c48766"} Jul 15 22:34:56.099651 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:56.099587 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zc8w9" podStartSLOduration=1.989366763 podStartE2EDuration="3.099565602s" totalImagesPullingTime="1.039009917s" totalInitContainerRuntime="71.188922ms" isStatefulPod=true podCreationTimestamp="2026-07-15 22:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:34:56.097589961 +0000 UTC m=+107.037792512" watchObservedRunningTime="2026-07-15 22:34:56.099565602 +0000 UTC m=+107.039768153" Jul 15 22:34:56.156022 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:56.155987 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:56.308355 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:56.308267 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:56.437560 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:56.437527 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:56.669251 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:56.669215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:57.357571 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:57.357531 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:57.473180 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:57.473128 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:58.162510 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:58.162476 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:58.550293 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:58.550210 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:58.671829 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:58.671791 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:34:59.520374 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:34:59.520336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:35:00.407958 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:35:00.407923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:35:01.589607 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:35:01.589572 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:35:04.103559 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:35:04.103528 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:35:04.201987 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:35:04.201951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:35:16.901227 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:35:16.901186 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:35:17.057906 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:35:17.057870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:35:25.409293 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:35:25.409255 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:35:25.613434 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:35:25.613401 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:09.230348 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:09.230254 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:09.295421 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:09.295378 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:12.838781 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:12.838711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:14.939663 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:14.939629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:21.093442 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:21.093402 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:21.217754 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:21.217716 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:23.975343 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:23.975303 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:24.028423 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:24.028387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:24.080990 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:24.080958 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:25.321586 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:25.321545 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:25.418750 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:25.418712 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:47.709586 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:47.709550 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:54.143629 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:54.143592 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:36:54.198919 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:36:54.198889 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:10.504740 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:10.504705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:10.590250 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:10.590220 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:32.229304 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:32.229217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:33.022995 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:33.022958 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:35.814670 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:35.814630 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:35.916966 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:35.916925 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:37.528054 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:37.528021 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:37.646840 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:37.646806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:37.769846 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:37.769815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:40.081058 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:40.081019 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:40.345141 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:40.345048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:42.609645 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:42.609605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:42.710189 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:42.710140 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:42.918992 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:42.918954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:43.019574 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:43.019543 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:47.878774 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:47.878724 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:49.491523 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:49.491489 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:49.591830 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:49.591798 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:53.629214 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:53.629179 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:53.727282 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:53.727249 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:37:55.313977 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.313940 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx"] Jul 15 22:37:55.316979 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.316961 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" Jul 15 22:37:55.320708 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.320687 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 22:37:55.320708 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.320696 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 22:37:55.320708 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.320706 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pzbcl\"" Jul 15 22:37:55.330371 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.330347 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx"] Jul 15 22:37:55.333456 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.333436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a5e4284-d779-4938-b80f-60e59c807fc9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24dtx\" (UID: \"1a5e4284-d779-4938-b80f-60e59c807fc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" Jul 15 22:37:55.333555 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.333481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhr8s\" (UniqueName: \"kubernetes.io/projected/1a5e4284-d779-4938-b80f-60e59c807fc9-kube-api-access-qhr8s\") pod \"openshift-lws-operator-fd8ccff4c-24dtx\" (UID: \"1a5e4284-d779-4938-b80f-60e59c807fc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" Jul 15 22:37:55.434381 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.434327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhr8s\" (UniqueName: \"kubernetes.io/projected/1a5e4284-d779-4938-b80f-60e59c807fc9-kube-api-access-qhr8s\") pod \"openshift-lws-operator-fd8ccff4c-24dtx\" (UID: \"1a5e4284-d779-4938-b80f-60e59c807fc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" Jul 15 22:37:55.434588 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.434429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a5e4284-d779-4938-b80f-60e59c807fc9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24dtx\" (UID: \"1a5e4284-d779-4938-b80f-60e59c807fc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" Jul 15 22:37:55.434826 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.434804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a5e4284-d779-4938-b80f-60e59c807fc9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24dtx\" (UID: \"1a5e4284-d779-4938-b80f-60e59c807fc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" Jul 15 22:37:55.444573 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.444548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhr8s\" (UniqueName: \"kubernetes.io/projected/1a5e4284-d779-4938-b80f-60e59c807fc9-kube-api-access-qhr8s\") pod \"openshift-lws-operator-fd8ccff4c-24dtx\" (UID: \"1a5e4284-d779-4938-b80f-60e59c807fc9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" Jul 15 22:37:55.626452 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.626363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" Jul 15 22:37:55.758017 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:55.757985 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx"] Jul 15 22:37:55.761354 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:37:55.761325 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a5e4284_d779_4938_b80f_60e59c807fc9.slice/crio-f70ef75ed14e73708eb1ca67cef9df144134f045e2c06ec5cbfb7f93e1f5efe4 WatchSource:0}: Error finding container f70ef75ed14e73708eb1ca67cef9df144134f045e2c06ec5cbfb7f93e1f5efe4: Status 404 returned error can't find the container with id f70ef75ed14e73708eb1ca67cef9df144134f045e2c06ec5cbfb7f93e1f5efe4 Jul 15 22:37:56.661159 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:56.661103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" event={"ID":"1a5e4284-d779-4938-b80f-60e59c807fc9","Type":"ContainerStarted","Data":"f70ef75ed14e73708eb1ca67cef9df144134f045e2c06ec5cbfb7f93e1f5efe4"} Jul 15 22:37:58.670682 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:58.670642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" event={"ID":"1a5e4284-d779-4938-b80f-60e59c807fc9","Type":"ContainerStarted","Data":"ff5cb2c5504d9c83d86c14ef6adde98b57db54fad80872de6527657cb4d1a241"} Jul 15 22:37:58.687753 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:58.687700 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24dtx" podStartSLOduration=1.330319625 podStartE2EDuration="3.68768437s" totalImagesPullingTime="2.357364745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:37:58.687551879 +0000 UTC m=+289.627754429" watchObservedRunningTime="2026-07-15 22:37:58.68768437 +0000 UTC m=+289.627886920" Jul 15 22:37:59.326525 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:37:59.326491 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:00.653632 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:00.653595 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:01.409366 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:01.409334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:09.485089 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:09.485055 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 22:38:13.971356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:13.971321 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:14.082207 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:14.082152 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:16.806947 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:16.806908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:16.918786 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:16.918755 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:17.441328 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.441296 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:17.717343 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.717258 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:17.836182 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.836142 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:17.893340 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.893311 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4"] Jul 15 22:38:17.899409 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.899382 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:17.905116 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.905090 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 22:38:17.905294 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.905192 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vzbsj\"" Jul 15 22:38:17.905374 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.905336 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 22:38:17.905600 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.905578 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 22:38:17.906043 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.905729 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 22:38:17.917184 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:17.917146 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4"] Jul 15 22:38:18.000158 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.000073 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c747a35-2e31-4a36-9794-f32e4f59b81e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.000158 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.000118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c747a35-2e31-4a36-9794-f32e4f59b81e-tmp\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.000158 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.000138 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxszk\" (UniqueName: \"kubernetes.io/projected/7c747a35-2e31-4a36-9794-f32e4f59b81e-kube-api-access-gxszk\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.000392 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.000248 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c747a35-2e31-4a36-9794-f32e4f59b81e-manifests\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.000392 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.000300 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c747a35-2e31-4a36-9794-f32e4f59b81e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.038853 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.038815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:18.101498 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.101459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c747a35-2e31-4a36-9794-f32e4f59b81e-tmp\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.101498 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.101498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxszk\" (UniqueName: \"kubernetes.io/projected/7c747a35-2e31-4a36-9794-f32e4f59b81e-kube-api-access-gxszk\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.101676 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.101524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c747a35-2e31-4a36-9794-f32e4f59b81e-manifests\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.101676 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.101657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c747a35-2e31-4a36-9794-f32e4f59b81e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.101773 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.101755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c747a35-2e31-4a36-9794-f32e4f59b81e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.101961 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.101943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c747a35-2e31-4a36-9794-f32e4f59b81e-manifests\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.103978 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.103949 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c747a35-2e31-4a36-9794-f32e4f59b81e-tmp\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.104234 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.104215 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c747a35-2e31-4a36-9794-f32e4f59b81e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.104392 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.104372 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c747a35-2e31-4a36-9794-f32e4f59b81e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.110304 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.110280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxszk\" (UniqueName: \"kubernetes.io/projected/7c747a35-2e31-4a36-9794-f32e4f59b81e-kube-api-access-gxszk\") pod \"opendatahub-operator-controller-manager-7db6597b7c-rlbg4\" (UID: \"7c747a35-2e31-4a36-9794-f32e4f59b81e\") " pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.142013 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.141963 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:18.209305 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.209269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:18.240580 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.240543 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:18.347748 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.347720 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:18.348022 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.348003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4"] Jul 15 22:38:18.350641 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:38:18.350607 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c747a35_2e31_4a36_9794_f32e4f59b81e.slice/crio-2320f487decc57941d821c9b75105ecb67242e4e217a6a14beab22fa7460a981 WatchSource:0}: Error finding container 2320f487decc57941d821c9b75105ecb67242e4e217a6a14beab22fa7460a981: Status 404 returned error can't find the container with id 2320f487decc57941d821c9b75105ecb67242e4e217a6a14beab22fa7460a981 Jul 15 22:38:18.352365 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.352346 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 22:38:18.440490 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.440451 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:18.540615 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.540531 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:18.739039 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:18.739003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" event={"ID":"7c747a35-2e31-4a36-9794-f32e4f59b81e","Type":"ContainerStarted","Data":"2320f487decc57941d821c9b75105ecb67242e4e217a6a14beab22fa7460a981"} Jul 15 22:38:21.752252 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:21.752212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" event={"ID":"7c747a35-2e31-4a36-9794-f32e4f59b81e","Type":"ContainerStarted","Data":"607312d48e68b695df003429ffac526a11382a82192f87c2dfa307f2eb206c0b"} Jul 15 22:38:22.757459 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:22.757358 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c747a35-2e31-4a36-9794-f32e4f59b81e" containerID="607312d48e68b695df003429ffac526a11382a82192f87c2dfa307f2eb206c0b" exitCode=0 Jul 15 22:38:22.757459 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:22.757423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" event={"ID":"7c747a35-2e31-4a36-9794-f32e4f59b81e","Type":"ContainerDied","Data":"607312d48e68b695df003429ffac526a11382a82192f87c2dfa307f2eb206c0b"} Jul 15 22:38:23.762622 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:23.762584 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" event={"ID":"7c747a35-2e31-4a36-9794-f32e4f59b81e","Type":"ContainerStarted","Data":"6fd595ec4d85be53e3ba34e5d3a584497eaf8d2d1ddb18335965fb0e8bf55a45"} Jul 15 22:38:23.763049 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:23.762720 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:23.789779 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:23.789731 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" podStartSLOduration=2.2937018780000002 podStartE2EDuration="6.789714574s" totalImagesPullingTime="3.427852253s" totalInitContainerRuntime="1.068160443s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:38:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:38:23.788749854 +0000 UTC m=+314.728952405" watchObservedRunningTime="2026-07-15 22:38:23.789714574 +0000 UTC m=+314.729917122" Jul 15 22:38:34.768388 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:34.768357 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7db6597b7c-rlbg4" Jul 15 22:38:37.558707 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:37.558621 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:37.650780 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:37.650742 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:38.154911 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.154849 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:38.185971 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.185936 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk"] Jul 15 22:38:38.189361 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.189339 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.193823 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.193798 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 22:38:38.193984 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.193944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 22:38:38.194087 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.194069 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 22:38:38.194153 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.194109 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 22:38:38.211113 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.211084 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk"] Jul 15 22:38:38.259951 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.259905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrh45\" (UniqueName: \"kubernetes.io/projected/ad7132bc-cceb-4ae8-a87f-66043bd3a762-kube-api-access-lrh45\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.260151 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.260033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad7132bc-cceb-4ae8-a87f-66043bd3a762-tmp\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.260151 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.260059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7132bc-cceb-4ae8-a87f-66043bd3a762-tls-certs\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.360455 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.360421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad7132bc-cceb-4ae8-a87f-66043bd3a762-tmp\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.360455 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.360456 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7132bc-cceb-4ae8-a87f-66043bd3a762-tls-certs\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.360662 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.360487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrh45\" (UniqueName: \"kubernetes.io/projected/ad7132bc-cceb-4ae8-a87f-66043bd3a762-kube-api-access-lrh45\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.362918 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.362895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad7132bc-cceb-4ae8-a87f-66043bd3a762-tmp\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.363271 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.363247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7132bc-cceb-4ae8-a87f-66043bd3a762-tls-certs\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.371294 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.371265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:38.377426 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.377404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrh45\" (UniqueName: \"kubernetes.io/projected/ad7132bc-cceb-4ae8-a87f-66043bd3a762-kube-api-access-lrh45\") pod \"kube-auth-proxy-69cf9c9cdb-m84hk\" (UID: \"ad7132bc-cceb-4ae8-a87f-66043bd3a762\") " pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.411100 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.411014 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ffnjm"] Jul 15 22:38:38.414914 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.414893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:38.418963 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.418936 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wrhgx\"" Jul 15 22:38:38.424385 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.424364 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 22:38:38.425125 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.425099 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ffnjm"] Jul 15 22:38:38.498573 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.498533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" Jul 15 22:38:38.513095 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.512985 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:38.561704 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.561673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7f3924-3714-47c3-bab4-56a63584aaf7-cert\") pod \"odh-model-controller-7998c67dc4-ffnjm\" (UID: \"8f7f3924-3714-47c3-bab4-56a63584aaf7\") " pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:38.562117 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.561728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf9b8\" (UniqueName: \"kubernetes.io/projected/8f7f3924-3714-47c3-bab4-56a63584aaf7-kube-api-access-lf9b8\") pod \"odh-model-controller-7998c67dc4-ffnjm\" (UID: \"8f7f3924-3714-47c3-bab4-56a63584aaf7\") " pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:38.615944 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.615902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:38.652448 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.652288 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk"] Jul 15 22:38:38.655363 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:38:38.655332 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7132bc_cceb_4ae8_a87f_66043bd3a762.slice/crio-0b40dd9063b8ed7b16b7d6ffa30d1669462fce7db04413a5f5472c3fc0f37868 WatchSource:0}: Error finding container 0b40dd9063b8ed7b16b7d6ffa30d1669462fce7db04413a5f5472c3fc0f37868: Status 404 returned error can't find the container with id 0b40dd9063b8ed7b16b7d6ffa30d1669462fce7db04413a5f5472c3fc0f37868 Jul 15 22:38:38.662920 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.662850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7f3924-3714-47c3-bab4-56a63584aaf7-cert\") pod \"odh-model-controller-7998c67dc4-ffnjm\" (UID: \"8f7f3924-3714-47c3-bab4-56a63584aaf7\") " pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:38.662920 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.662902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9b8\" (UniqueName: \"kubernetes.io/projected/8f7f3924-3714-47c3-bab4-56a63584aaf7-kube-api-access-lf9b8\") pod \"odh-model-controller-7998c67dc4-ffnjm\" (UID: \"8f7f3924-3714-47c3-bab4-56a63584aaf7\") " pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:38.665810 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.665787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7f3924-3714-47c3-bab4-56a63584aaf7-cert\") pod \"odh-model-controller-7998c67dc4-ffnjm\" (UID: \"8f7f3924-3714-47c3-bab4-56a63584aaf7\") " pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:38.679317 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.679287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9b8\" (UniqueName: \"kubernetes.io/projected/8f7f3924-3714-47c3-bab4-56a63584aaf7-kube-api-access-lf9b8\") pod \"odh-model-controller-7998c67dc4-ffnjm\" (UID: \"8f7f3924-3714-47c3-bab4-56a63584aaf7\") " pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:38.712575 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.712537 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:38.726422 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.726376 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:38.810243 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.810205 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:38.814060 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.814029 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" event={"ID":"ad7132bc-cceb-4ae8-a87f-66043bd3a762","Type":"ContainerStarted","Data":"0b40dd9063b8ed7b16b7d6ffa30d1669462fce7db04413a5f5472c3fc0f37868"} Jul 15 22:38:38.895347 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:38.895299 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ffnjm"] Jul 15 22:38:38.898395 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:38:38.898361 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7f3924_3714_47c3_bab4_56a63584aaf7.slice/crio-72f8ff6e0ef40b777cdbc8da7ec4a2865370acd40c34375cfc2e257b6497f361 WatchSource:0}: Error finding container 72f8ff6e0ef40b777cdbc8da7ec4a2865370acd40c34375cfc2e257b6497f361: Status 404 returned error can't find the container with id 72f8ff6e0ef40b777cdbc8da7ec4a2865370acd40c34375cfc2e257b6497f361 Jul 15 22:38:39.473870 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.473833 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mzwkw"] Jul 15 22:38:39.477055 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.477035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:39.479956 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.479910 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 22:38:39.500086 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.500014 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mzwkw"] Jul 15 22:38:39.500653 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.500621 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:39.571148 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.571108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b6e213-1194-4025-a533-d426f495a78d-tls-certs\") pod \"model-serving-api-54d89bd5fc-mzwkw\" (UID: \"93b6e213-1194-4025-a533-d426f495a78d\") " pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:39.571610 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.571222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hwnx\" (UniqueName: \"kubernetes.io/projected/93b6e213-1194-4025-a533-d426f495a78d-kube-api-access-9hwnx\") pod \"model-serving-api-54d89bd5fc-mzwkw\" (UID: \"93b6e213-1194-4025-a533-d426f495a78d\") " pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:39.672198 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.671956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b6e213-1194-4025-a533-d426f495a78d-tls-certs\") pod \"model-serving-api-54d89bd5fc-mzwkw\" (UID: \"93b6e213-1194-4025-a533-d426f495a78d\") " pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:39.672198 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.672036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hwnx\" (UniqueName: \"kubernetes.io/projected/93b6e213-1194-4025-a533-d426f495a78d-kube-api-access-9hwnx\") pod \"model-serving-api-54d89bd5fc-mzwkw\" (UID: \"93b6e213-1194-4025-a533-d426f495a78d\") " pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:39.673188 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:38:39.672826 2565 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 15 22:38:39.673188 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:38:39.672895 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b6e213-1194-4025-a533-d426f495a78d-tls-certs podName:93b6e213-1194-4025-a533-d426f495a78d nodeName:}" failed. No retries permitted until 2026-07-15 22:38:40.172874022 +0000 UTC m=+331.113076566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/93b6e213-1194-4025-a533-d426f495a78d-tls-certs") pod "model-serving-api-54d89bd5fc-mzwkw" (UID: "93b6e213-1194-4025-a533-d426f495a78d") : secret "model-serving-api-tls" not found Jul 15 22:38:39.675533 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.675502 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:39.690922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.684095 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6vslm"] Jul 15 22:38:39.690922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.687547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.690922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.690302 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 22:38:39.690922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.690342 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 22:38:39.690922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.690568 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gf6g9\"" Jul 15 22:38:39.690922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.690754 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 22:38:39.690922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.690781 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 22:38:39.701861 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.701832 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6vslm"] Jul 15 22:38:39.705009 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.704952 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hwnx\" (UniqueName: \"kubernetes.io/projected/93b6e213-1194-4025-a533-d426f495a78d-kube-api-access-9hwnx\") pod \"model-serving-api-54d89bd5fc-mzwkw\" (UID: \"93b6e213-1194-4025-a533-d426f495a78d\") " pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:39.773489 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.773377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.774246 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.774012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xrg\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-kube-api-access-b5xrg\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.774246 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.774093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.774246 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.774148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.774776 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.774561 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ee3f66-f20c-45d7-93c9-c69a3ec04496-local-certs\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.774776 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.774593 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-cacerts\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.774776 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.774660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-token\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.780600 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.780081 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:39.821463 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.821413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" event={"ID":"8f7f3924-3714-47c3-bab4-56a63584aaf7","Type":"ContainerStarted","Data":"72f8ff6e0ef40b777cdbc8da7ec4a2865370acd40c34375cfc2e257b6497f361"} Jul 15 22:38:39.875911 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.875867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.876097 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.875925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xrg\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-kube-api-access-b5xrg\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.876097 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.875984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.876097 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.876040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.876097 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.876083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ee3f66-f20c-45d7-93c9-c69a3ec04496-local-certs\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.876355 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.876107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-cacerts\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.876828 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.876788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-token\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.877843 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.877819 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.879598 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.879548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.880056 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.880031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ee3f66-f20c-45d7-93c9-c69a3ec04496-local-certs\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.881156 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.881131 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-cacerts\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.883754 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.883703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.888351 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.888329 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-token\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:39.892934 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:39.892893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xrg\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-kube-api-access-b5xrg\") pod \"istiod-openshift-gateway-7574459656-6vslm\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:40.004487 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.004140 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:40.137696 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.137568 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:40.178838 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.178792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b6e213-1194-4025-a533-d426f495a78d-tls-certs\") pod \"model-serving-api-54d89bd5fc-mzwkw\" (UID: \"93b6e213-1194-4025-a533-d426f495a78d\") " pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:40.182902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.182868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b6e213-1194-4025-a533-d426f495a78d-tls-certs\") pod \"model-serving-api-54d89bd5fc-mzwkw\" (UID: \"93b6e213-1194-4025-a533-d426f495a78d\") " pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:40.193131 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.191737 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6vslm"] Jul 15 22:38:40.195879 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:38:40.195846 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ee3f66_f20c_45d7_93c9_c69a3ec04496.slice/crio-615f22ee7a0a743d6bb015513fe6244438f8932d7364d84bf439aa346e4a1528 WatchSource:0}: Error finding container 615f22ee7a0a743d6bb015513fe6244438f8932d7364d84bf439aa346e4a1528: Status 404 returned error can't find the container with id 615f22ee7a0a743d6bb015513fe6244438f8932d7364d84bf439aa346e4a1528 Jul 15 22:38:40.388737 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.388435 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:40.577119 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.577044 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mzwkw"] Jul 15 22:38:40.581199 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:38:40.581137 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b6e213_1194_4025_a533_d426f495a78d.slice/crio-05c9505a0719a66978f08d4178111b237465f904f68f5c8e6a8441d044fdb4df WatchSource:0}: Error finding container 05c9505a0719a66978f08d4178111b237465f904f68f5c8e6a8441d044fdb4df: Status 404 returned error can't find the container with id 05c9505a0719a66978f08d4178111b237465f904f68f5c8e6a8441d044fdb4df Jul 15 22:38:40.679109 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.679002 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:40.828453 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.828404 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" event={"ID":"93b6e213-1194-4025-a533-d426f495a78d","Type":"ContainerStarted","Data":"05c9505a0719a66978f08d4178111b237465f904f68f5c8e6a8441d044fdb4df"} Jul 15 22:38:40.830126 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:40.830089 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" event={"ID":"54ee3f66-f20c-45d7-93c9-c69a3ec04496","Type":"ContainerStarted","Data":"615f22ee7a0a743d6bb015513fe6244438f8932d7364d84bf439aa346e4a1528"} Jul 15 22:38:42.036775 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:42.036741 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:44.460862 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.460826 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 22:38:44.461132 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.460888 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 15 22:38:44.852045 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.852000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" event={"ID":"8f7f3924-3714-47c3-bab4-56a63584aaf7","Type":"ContainerStarted","Data":"bf77ff1067167f85a03ecc91b3b9882c59334c683e64ac5824d3616a80b38849"} Jul 15 22:38:44.852494 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.852463 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:38:44.854372 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.854336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" event={"ID":"54ee3f66-f20c-45d7-93c9-c69a3ec04496","Type":"ContainerStarted","Data":"6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b"} Jul 15 22:38:44.854505 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.854429 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:44.856456 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.856429 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" event={"ID":"ad7132bc-cceb-4ae8-a87f-66043bd3a762","Type":"ContainerStarted","Data":"f8798b9faeae21d27e849830ae281d4f9283ce34343c4ea889b692e01f060071"} Jul 15 22:38:44.858094 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.858068 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" event={"ID":"93b6e213-1194-4025-a533-d426f495a78d","Type":"ContainerStarted","Data":"1d993d35ec35a3c96414b063a69896aebebd1f83b3e55d782e6106eafffad644"} Jul 15 22:38:44.858250 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.858232 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:44.870969 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.870922 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" podStartSLOduration=1.374089108 podStartE2EDuration="6.870908238s" totalImagesPullingTime="5.49681913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:38:44.869177895 +0000 UTC m=+335.809380438" watchObservedRunningTime="2026-07-15 22:38:44.870908238 +0000 UTC m=+335.811110788" Jul 15 22:38:44.887374 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.887325 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" podStartSLOduration=2.070398771 podStartE2EDuration="5.887316127s" totalImagesPullingTime="3.816917356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:38:44.886693344 +0000 UTC m=+335.826895895" watchObservedRunningTime="2026-07-15 22:38:44.887316127 +0000 UTC m=+335.827518676" Jul 15 22:38:44.906430 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:44.906369 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" podStartSLOduration=1.643801388 podStartE2EDuration="5.906349715s" totalImagesPullingTime="4.262548327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:38:44.905797242 +0000 UTC m=+335.845999795" watchObservedRunningTime="2026-07-15 22:38:44.906349715 +0000 UTC m=+335.846552267" Jul 15 22:38:45.180652 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:45.180618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:45.284353 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:45.284315 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:45.808982 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:45.807793 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:45.864420 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:45.864386 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-6vslm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 22:38:45.864583 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:45.864459 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" podUID="54ee3f66-f20c-45d7-93c9-c69a3ec04496" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 22:38:45.903211 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:45.903159 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:46.052254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:46.052225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:48.863899 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:48.863865 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:48.890205 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:48.890134 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69cf9c9cdb-m84hk" podStartSLOduration=5.150690428 podStartE2EDuration="10.890117575s" totalImagesPullingTime="5.739427147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:38:44.923758878 +0000 UTC m=+335.863961428" watchObservedRunningTime="2026-07-15 22:38:48.890117575 +0000 UTC m=+339.830320125" Jul 15 22:38:49.902108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.902060 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:49.971755 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.971719 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6"] Jul 15 22:38:49.974806 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.974779 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:49.974946 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.974847 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7"] Jul 15 22:38:49.977378 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.977352 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 22:38:49.977872 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.977855 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-695d7\"" Jul 15 22:38:49.977935 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.977914 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:49.980493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.980469 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 22:38:49.983743 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:49.983725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mjfrn\"" Jul 15 22:38:50.002201 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.002153 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7"] Jul 15 22:38:50.003835 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.003743 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6"] Jul 15 22:38:50.016810 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.016776 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:50.070788 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.070752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c5fa0f1-2df0-449f-9b57-ca787b186d2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-lk4z6\" (UID: \"5c5fa0f1-2df0-449f-9b57-ca787b186d2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:50.070976 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.070798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36d6a42b-91be-41e8-9286-42e55349e31f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8rkb7\" (UID: \"36d6a42b-91be-41e8-9286-42e55349e31f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:50.070976 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.070829 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgh4l\" (UniqueName: \"kubernetes.io/projected/36d6a42b-91be-41e8-9286-42e55349e31f-kube-api-access-cgh4l\") pod \"llmisvc-controller-manager-777c6bd7d5-8rkb7\" (UID: \"36d6a42b-91be-41e8-9286-42e55349e31f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:50.070976 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.070898 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9sz\" (UniqueName: \"kubernetes.io/projected/5c5fa0f1-2df0-449f-9b57-ca787b186d2d-kube-api-access-sz9sz\") pod \"kserve-controller-manager-d6f4b65b8-lk4z6\" (UID: \"5c5fa0f1-2df0-449f-9b57-ca787b186d2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:50.151514 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.151481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:50.171746 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.171674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36d6a42b-91be-41e8-9286-42e55349e31f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8rkb7\" (UID: \"36d6a42b-91be-41e8-9286-42e55349e31f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:50.171746 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.171724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgh4l\" (UniqueName: \"kubernetes.io/projected/36d6a42b-91be-41e8-9286-42e55349e31f-kube-api-access-cgh4l\") pod \"llmisvc-controller-manager-777c6bd7d5-8rkb7\" (UID: \"36d6a42b-91be-41e8-9286-42e55349e31f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:50.171926 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.171839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9sz\" (UniqueName: \"kubernetes.io/projected/5c5fa0f1-2df0-449f-9b57-ca787b186d2d-kube-api-access-sz9sz\") pod \"kserve-controller-manager-d6f4b65b8-lk4z6\" (UID: \"5c5fa0f1-2df0-449f-9b57-ca787b186d2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:50.171926 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.171904 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c5fa0f1-2df0-449f-9b57-ca787b186d2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-lk4z6\" (UID: \"5c5fa0f1-2df0-449f-9b57-ca787b186d2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:50.174447 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.174421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36d6a42b-91be-41e8-9286-42e55349e31f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8rkb7\" (UID: \"36d6a42b-91be-41e8-9286-42e55349e31f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:50.174569 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.174477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c5fa0f1-2df0-449f-9b57-ca787b186d2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-lk4z6\" (UID: \"5c5fa0f1-2df0-449f-9b57-ca787b186d2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:50.187214 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.187188 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9sz\" (UniqueName: \"kubernetes.io/projected/5c5fa0f1-2df0-449f-9b57-ca787b186d2d-kube-api-access-sz9sz\") pod \"kserve-controller-manager-d6f4b65b8-lk4z6\" (UID: \"5c5fa0f1-2df0-449f-9b57-ca787b186d2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:50.187327 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.187308 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgh4l\" (UniqueName: \"kubernetes.io/projected/36d6a42b-91be-41e8-9286-42e55349e31f-kube-api-access-cgh4l\") pod \"llmisvc-controller-manager-777c6bd7d5-8rkb7\" (UID: \"36d6a42b-91be-41e8-9286-42e55349e31f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:50.252693 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.252659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:50.287784 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.287753 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:50.292873 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.292848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:50.463246 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.463222 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6"] Jul 15 22:38:50.464660 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:38:50.464635 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5fa0f1_2df0_449f_9b57_ca787b186d2d.slice/crio-24809c368a81adf46335751797cc271e7520f6359ba145e84438b442322ac79b WatchSource:0}: Error finding container 24809c368a81adf46335751797cc271e7520f6359ba145e84438b442322ac79b: Status 404 returned error can't find the container with id 24809c368a81adf46335751797cc271e7520f6359ba145e84438b442322ac79b Jul 15 22:38:50.502978 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.502945 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7"] Jul 15 22:38:50.505681 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:38:50.505655 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod36d6a42b_91be_41e8_9286_42e55349e31f.slice/crio-b2f6c9d90185621dcd0712c53dda92b253f49a5a8f9f287c3a2d1204d79a1eee WatchSource:0}: Error finding container b2f6c9d90185621dcd0712c53dda92b253f49a5a8f9f287c3a2d1204d79a1eee: Status 404 returned error can't find the container with id b2f6c9d90185621dcd0712c53dda92b253f49a5a8f9f287c3a2d1204d79a1eee Jul 15 22:38:50.883444 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.883411 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" event={"ID":"36d6a42b-91be-41e8-9286-42e55349e31f","Type":"ContainerStarted","Data":"b2f6c9d90185621dcd0712c53dda92b253f49a5a8f9f287c3a2d1204d79a1eee"} Jul 15 22:38:50.884557 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:50.884537 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" event={"ID":"5c5fa0f1-2df0-449f-9b57-ca787b186d2d","Type":"ContainerStarted","Data":"24809c368a81adf46335751797cc271e7520f6359ba145e84438b442322ac79b"} Jul 15 22:38:51.213607 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:51.213520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:51.312449 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:51.312420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:51.413322 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:51.413291 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:52.562811 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:52.562780 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6vslm"] Jul 15 22:38:52.563331 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:52.563004 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" podUID="54ee3f66-f20c-45d7-93c9-c69a3ec04496" containerName="discovery" containerID="cri-o://6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b" gracePeriod=30 Jul 15 22:38:53.726518 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.726491 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:53.801091 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.801060 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-kubeconfig\") pod \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " Jul 15 22:38:53.801273 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.801100 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-dns-cert\") pod \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " Jul 15 22:38:53.801273 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.801131 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5xrg\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-kube-api-access-b5xrg\") pod \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " Jul 15 22:38:53.801273 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.801161 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-token\") pod \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " Jul 15 22:38:53.801420 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.801348 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-cacerts\") pod \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " Jul 15 22:38:53.801474 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.801434 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ee3f66-f20c-45d7-93c9-c69a3ec04496-local-certs\") pod \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " Jul 15 22:38:53.801527 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.801473 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-ca-configmap\") pod \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\" (UID: \"54ee3f66-f20c-45d7-93c9-c69a3ec04496\") " Jul 15 22:38:53.802081 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.802050 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "54ee3f66-f20c-45d7-93c9-c69a3ec04496" (UID: "54ee3f66-f20c-45d7-93c9-c69a3ec04496"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 22:38:53.804701 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.804600 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-token" (OuterVolumeSpecName: "istio-token") pod "54ee3f66-f20c-45d7-93c9-c69a3ec04496" (UID: "54ee3f66-f20c-45d7-93c9-c69a3ec04496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:38:53.804701 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.804679 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "54ee3f66-f20c-45d7-93c9-c69a3ec04496" (UID: "54ee3f66-f20c-45d7-93c9-c69a3ec04496"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:38:53.804701 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.804697 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ee3f66-f20c-45d7-93c9-c69a3ec04496-local-certs" (OuterVolumeSpecName: "local-certs") pod "54ee3f66-f20c-45d7-93c9-c69a3ec04496" (UID: "54ee3f66-f20c-45d7-93c9-c69a3ec04496"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:38:53.804945 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.804925 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "54ee3f66-f20c-45d7-93c9-c69a3ec04496" (UID: "54ee3f66-f20c-45d7-93c9-c69a3ec04496"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:38:53.805330 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.805308 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-cacerts" (OuterVolumeSpecName: "cacerts") pod "54ee3f66-f20c-45d7-93c9-c69a3ec04496" (UID: "54ee3f66-f20c-45d7-93c9-c69a3ec04496"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:38:53.805748 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.805728 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-kube-api-access-b5xrg" (OuterVolumeSpecName: "kube-api-access-b5xrg") pod "54ee3f66-f20c-45d7-93c9-c69a3ec04496" (UID: "54ee3f66-f20c-45d7-93c9-c69a3ec04496"). InnerVolumeSpecName "kube-api-access-b5xrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:38:53.901476 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.901439 2565 generic.go:356] "Generic (PLEG): container finished" podID="54ee3f66-f20c-45d7-93c9-c69a3ec04496" containerID="6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b" exitCode=0 Jul 15 22:38:53.901639 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.901498 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" Jul 15 22:38:53.901639 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.901520 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" event={"ID":"54ee3f66-f20c-45d7-93c9-c69a3ec04496","Type":"ContainerDied","Data":"6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b"} Jul 15 22:38:53.901639 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.901611 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6vslm" event={"ID":"54ee3f66-f20c-45d7-93c9-c69a3ec04496","Type":"ContainerDied","Data":"615f22ee7a0a743d6bb015513fe6244438f8932d7364d84bf439aa346e4a1528"} Jul 15 22:38:53.901639 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.901638 2565 scope.go:117] "RemoveContainer" containerID="6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b" Jul 15 22:38:53.902098 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.902074 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-cacerts\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:38:53.902098 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.902097 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ee3f66-f20c-45d7-93c9-c69a3ec04496-local-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:38:53.902300 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.902109 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-ca-configmap\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:38:53.902300 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.902122 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-kubeconfig\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:38:53.902300 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.902135 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-csr-dns-cert\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:38:53.902300 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.902147 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xrg\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-kube-api-access-b5xrg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:38:53.902300 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.902159 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ee3f66-f20c-45d7-93c9-c69a3ec04496-istio-token\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:38:53.903233 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.903203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" event={"ID":"5c5fa0f1-2df0-449f-9b57-ca787b186d2d","Type":"ContainerStarted","Data":"910f3025bf006f307998a74f1f6de8f837c59c0ea216c1a12a9ae326513305eb"} Jul 15 22:38:53.903356 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.903302 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:38:53.921826 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.921803 2565 scope.go:117] "RemoveContainer" containerID="6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b" Jul 15 22:38:53.922222 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:38:53.922196 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b\": container with ID starting with 6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b not found: ID does not exist" containerID="6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b" Jul 15 22:38:53.922339 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.922230 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b"} err="failed to get container status \"6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b\": rpc error: code = NotFound desc = could not find container \"6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b\": container with ID starting with 6498cb65bf581c2d96a8c253307e9037a83d6903edb5f367d67a26846fa63f2b not found: ID does not exist" Jul 15 22:38:53.927274 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.927226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" podStartSLOduration=2.307452387 podStartE2EDuration="4.927211413s" totalImagesPullingTime="2.619759026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:38:53.925902056 +0000 UTC m=+344.866104608" watchObservedRunningTime="2026-07-15 22:38:53.927211413 +0000 UTC m=+344.867413963" Jul 15 22:38:53.941159 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.941136 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6vslm"] Jul 15 22:38:53.945463 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:53.945441 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6vslm"] Jul 15 22:38:54.909360 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:54.909323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" event={"ID":"36d6a42b-91be-41e8-9286-42e55349e31f","Type":"ContainerStarted","Data":"56aa5710b4f9bb970cd5999cf34bba3caf9fa56f59f15e517a57efba30667e23"} Jul 15 22:38:54.909790 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:54.909423 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:38:54.928412 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:54.928367 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" podStartSLOduration=2.672382237 podStartE2EDuration="5.928351481s" totalImagesPullingTime="3.255969244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:38:54.928045483 +0000 UTC m=+345.868248031" watchObservedRunningTime="2026-07-15 22:38:54.928351481 +0000 UTC m=+345.868554031" Jul 15 22:38:55.346946 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:55.346859 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:55.450488 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:55.450453 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:38:55.584653 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:55.584616 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ee3f66-f20c-45d7-93c9-c69a3ec04496" path="/var/lib/kubelet/pods/54ee3f66-f20c-45d7-93c9-c69a3ec04496/volumes" Jul 15 22:38:55.866335 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:55.866308 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mzwkw" Jul 15 22:38:55.895535 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:38:55.895502 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ffnjm" Jul 15 22:39:03.743209 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:03.743101 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:39:24.916431 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:24.916394 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lk4z6" Jul 15 22:39:25.917885 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:25.917855 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8rkb7" Jul 15 22:39:29.404422 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:29.404387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:39:38.952764 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:38.952725 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:39:39.052987 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.052955 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:39:39.087422 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.087389 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tzqh"] Jul 15 22:39:39.087685 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.087673 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ee3f66-f20c-45d7-93c9-c69a3ec04496" containerName="discovery" Jul 15 22:39:39.087685 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.087686 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ee3f66-f20c-45d7-93c9-c69a3ec04496" containerName="discovery" Jul 15 22:39:39.087763 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.087737 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ee3f66-f20c-45d7-93c9-c69a3ec04496" containerName="discovery" Jul 15 22:39:39.090741 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.090718 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" Jul 15 22:39:39.093553 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.093454 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 22:39:39.093553 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.093497 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-p8t4n\"" Jul 15 22:39:39.093553 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.093468 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 22:39:39.100145 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.100121 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tzqh"] Jul 15 22:39:39.144871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.144838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8dv6\" (UniqueName: \"kubernetes.io/projected/ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f-kube-api-access-c8dv6\") pod \"kuadrant-operator-catalog-6tzqh\" (UID: \"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f\") " pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" Jul 15 22:39:39.151838 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.151809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:39:39.245548 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.245462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8dv6\" (UniqueName: \"kubernetes.io/projected/ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f-kube-api-access-c8dv6\") pod \"kuadrant-operator-catalog-6tzqh\" (UID: \"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f\") " pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" Jul 15 22:39:39.253735 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.253703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8dv6\" (UniqueName: \"kubernetes.io/projected/ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f-kube-api-access-c8dv6\") pod \"kuadrant-operator-catalog-6tzqh\" (UID: \"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f\") " pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" Jul 15 22:39:39.402705 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.402666 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" Jul 15 22:39:39.445702 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.445669 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tzqh"] Jul 15 22:39:39.556152 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.556114 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tzqh"] Jul 15 22:39:39.560343 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:39:39.560315 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3f4f2b_ca02_44d9_9abf_3c4c1f17264f.slice/crio-fd1622bcaf8513e50087a48b3a084c955039eba9edb601983e17e527ce304aca WatchSource:0}: Error finding container fd1622bcaf8513e50087a48b3a084c955039eba9edb601983e17e527ce304aca: Status 404 returned error can't find the container with id fd1622bcaf8513e50087a48b3a084c955039eba9edb601983e17e527ce304aca Jul 15 22:39:39.653043 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.653012 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmh75"] Jul 15 22:39:39.657154 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.657137 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:39.663882 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.663856 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmh75"] Jul 15 22:39:39.749801 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.749763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt566\" (UniqueName: \"kubernetes.io/projected/47708cb9-7dff-4044-97f0-125a8ba74428-kube-api-access-dt566\") pod \"kuadrant-operator-catalog-wmh75\" (UID: \"47708cb9-7dff-4044-97f0-125a8ba74428\") " pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:39.850805 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.850717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt566\" (UniqueName: \"kubernetes.io/projected/47708cb9-7dff-4044-97f0-125a8ba74428-kube-api-access-dt566\") pod \"kuadrant-operator-catalog-wmh75\" (UID: \"47708cb9-7dff-4044-97f0-125a8ba74428\") " pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:39.859887 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.859860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt566\" (UniqueName: \"kubernetes.io/projected/47708cb9-7dff-4044-97f0-125a8ba74428-kube-api-access-dt566\") pod \"kuadrant-operator-catalog-wmh75\" (UID: \"47708cb9-7dff-4044-97f0-125a8ba74428\") " pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:39.879854 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.879824 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:39:39.967651 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:39.967616 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:40.085063 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:40.085028 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:39:40.092125 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:40.092091 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" event={"ID":"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f","Type":"ContainerStarted","Data":"fd1622bcaf8513e50087a48b3a084c955039eba9edb601983e17e527ce304aca"} Jul 15 22:39:40.105096 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:40.104916 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmh75"] Jul 15 22:39:40.107602 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:39:40.107573 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47708cb9_7dff_4044_97f0_125a8ba74428.slice/crio-84dd7f455fa83472b2b04706808e658dae2d6d0bd1cb7101187f1a5d7f35c615 WatchSource:0}: Error finding container 84dd7f455fa83472b2b04706808e658dae2d6d0bd1cb7101187f1a5d7f35c615: Status 404 returned error can't find the container with id 84dd7f455fa83472b2b04706808e658dae2d6d0bd1cb7101187f1a5d7f35c615 Jul 15 22:39:41.098291 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:41.098246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" event={"ID":"47708cb9-7dff-4044-97f0-125a8ba74428","Type":"ContainerStarted","Data":"84dd7f455fa83472b2b04706808e658dae2d6d0bd1cb7101187f1a5d7f35c615"} Jul 15 22:39:42.104311 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.104211 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" event={"ID":"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f","Type":"ContainerStarted","Data":"6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1"} Jul 15 22:39:42.104742 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.104313 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" podUID="ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f" containerName="registry-server" containerID="cri-o://6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1" gracePeriod=2 Jul 15 22:39:42.105801 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.105770 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" event={"ID":"47708cb9-7dff-4044-97f0-125a8ba74428","Type":"ContainerStarted","Data":"27d8ef0602d431b20ef4a77dc54ab21b3e91e4f2a94ed0b1ecb96320a0f5b85e"} Jul 15 22:39:42.120825 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.120777 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" podStartSLOduration=0.945564745 podStartE2EDuration="3.120762222s" totalImagesPullingTime="2.175197477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:39:42.119246905 +0000 UTC m=+393.059449456" watchObservedRunningTime="2026-07-15 22:39:42.120762222 +0000 UTC m=+393.060964772" Jul 15 22:39:42.134952 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.134905 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" podStartSLOduration=1.530382802 podStartE2EDuration="3.134889522s" totalImagesPullingTime="1.60450672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:39:42.134265191 +0000 UTC m=+393.074467742" watchObservedRunningTime="2026-07-15 22:39:42.134889522 +0000 UTC m=+393.075092106" Jul 15 22:39:42.346272 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.346244 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" Jul 15 22:39:42.371854 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.371764 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8dv6\" (UniqueName: \"kubernetes.io/projected/ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f-kube-api-access-c8dv6\") pod \"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f\" (UID: \"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f\") " Jul 15 22:39:42.374258 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.374227 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f-kube-api-access-c8dv6" (OuterVolumeSpecName: "kube-api-access-c8dv6") pod "ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f" (UID: "ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f"). InnerVolumeSpecName "kube-api-access-c8dv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:39:42.473023 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:42.472982 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8dv6\" (UniqueName: \"kubernetes.io/projected/ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f-kube-api-access-c8dv6\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:39:43.110962 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.110873 2565 generic.go:356] "Generic (PLEG): container finished" podID="ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f" containerID="6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1" exitCode=0 Jul 15 22:39:43.110962 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.110915 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" Jul 15 22:39:43.111429 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.110958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" event={"ID":"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f","Type":"ContainerDied","Data":"6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1"} Jul 15 22:39:43.111429 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.111007 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6tzqh" event={"ID":"ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f","Type":"ContainerDied","Data":"fd1622bcaf8513e50087a48b3a084c955039eba9edb601983e17e527ce304aca"} Jul 15 22:39:43.111429 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.111029 2565 scope.go:117] "RemoveContainer" containerID="6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1" Jul 15 22:39:43.119849 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.119831 2565 scope.go:117] "RemoveContainer" containerID="6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1" Jul 15 22:39:43.120104 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:39:43.120086 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1\": container with ID starting with 6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1 not found: ID does not exist" containerID="6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1" Jul 15 22:39:43.120154 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.120113 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1"} err="failed to get container status \"6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1\": rpc error: code = NotFound desc = could not find container \"6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1\": container with ID starting with 6849e66674429fbfd19140dcf419771a2a0e9e337efca43531e111b88bd0c0e1 not found: ID does not exist" Jul 15 22:39:43.135376 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.135348 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tzqh"] Jul 15 22:39:43.139561 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.139532 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tzqh"] Jul 15 22:39:43.584763 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:43.584727 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f" path="/var/lib/kubelet/pods/ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f/volumes" Jul 15 22:39:49.968581 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:49.968540 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:49.969000 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:49.968602 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:49.991478 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:49.991445 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:50.162476 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:50.162445 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wmh75" Jul 15 22:39:52.882086 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:39:52.882050 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:06.866631 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:06.866594 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:07.009259 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:07.009224 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:07.111081 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:07.111027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:08.059488 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:08.059451 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:08.158408 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:08.158381 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:11.379846 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:11.379803 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:12.649003 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.648973 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd"] Jul 15 22:40:12.649399 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.649295 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f" containerName="registry-server" Jul 15 22:40:12.649399 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.649307 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f" containerName="registry-server" Jul 15 22:40:12.649399 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.649373 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab3f4f2b-ca02-44d9-9abf-3c4c1f17264f" containerName="registry-server" Jul 15 22:40:12.657423 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.657400 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:12.660181 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.660141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wt2v9\"" Jul 15 22:40:12.669475 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.669446 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd"] Jul 15 22:40:12.729977 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.729939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9423f4f-8eee-4573-9c89-a540028e97ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" (UID: \"d9423f4f-8eee-4573-9c89-a540028e97ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:12.730149 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.730008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg4pr\" (UniqueName: \"kubernetes.io/projected/d9423f4f-8eee-4573-9c89-a540028e97ce-kube-api-access-kg4pr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" (UID: \"d9423f4f-8eee-4573-9c89-a540028e97ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:12.831242 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.831157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg4pr\" (UniqueName: \"kubernetes.io/projected/d9423f4f-8eee-4573-9c89-a540028e97ce-kube-api-access-kg4pr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" (UID: \"d9423f4f-8eee-4573-9c89-a540028e97ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:12.831428 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.831323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9423f4f-8eee-4573-9c89-a540028e97ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" (UID: \"d9423f4f-8eee-4573-9c89-a540028e97ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:12.831742 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.831720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9423f4f-8eee-4573-9c89-a540028e97ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" (UID: \"d9423f4f-8eee-4573-9c89-a540028e97ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:12.843647 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.843611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg4pr\" (UniqueName: \"kubernetes.io/projected/d9423f4f-8eee-4573-9c89-a540028e97ce-kube-api-access-kg4pr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" (UID: \"d9423f4f-8eee-4573-9c89-a540028e97ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:12.967776 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:12.967680 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:13.104129 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:13.104089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd"] Jul 15 22:40:13.111251 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:40:13.111205 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9423f4f_8eee_4573_9c89_a540028e97ce.slice/crio-45b7b119583e7452b31f24c9fe984af4aaac99395d50cbf580643bbe0924a41e WatchSource:0}: Error finding container 45b7b119583e7452b31f24c9fe984af4aaac99395d50cbf580643bbe0924a41e: Status 404 returned error can't find the container with id 45b7b119583e7452b31f24c9fe984af4aaac99395d50cbf580643bbe0924a41e Jul 15 22:40:13.232184 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:13.232082 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" event={"ID":"d9423f4f-8eee-4573-9c89-a540028e97ce","Type":"ContainerStarted","Data":"45b7b119583e7452b31f24c9fe984af4aaac99395d50cbf580643bbe0924a41e"} Jul 15 22:40:13.334038 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:13.333994 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:14.638670 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.638630 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45"] Jul 15 22:40:14.642698 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.642664 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:14.645467 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.645439 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xn65j\"" Jul 15 22:40:14.654302 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.654246 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45"] Jul 15 22:40:14.746591 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.746551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5grf\" (UniqueName: \"kubernetes.io/projected/c9b13014-57d9-4900-9916-50a147ff7f77-kube-api-access-p5grf\") pod \"limitador-operator-controller-manager-859d7b78bd-pxx45\" (UID: \"c9b13014-57d9-4900-9916-50a147ff7f77\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:14.847595 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.847561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5grf\" (UniqueName: \"kubernetes.io/projected/c9b13014-57d9-4900-9916-50a147ff7f77-kube-api-access-p5grf\") pod \"limitador-operator-controller-manager-859d7b78bd-pxx45\" (UID: \"c9b13014-57d9-4900-9916-50a147ff7f77\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:14.857214 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.857157 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5grf\" (UniqueName: \"kubernetes.io/projected/c9b13014-57d9-4900-9916-50a147ff7f77-kube-api-access-p5grf\") pod \"limitador-operator-controller-manager-859d7b78bd-pxx45\" (UID: \"c9b13014-57d9-4900-9916-50a147ff7f77\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:14.915206 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.915079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:14.956226 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:14.956188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:15.102050 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:15.101954 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45"] Jul 15 22:40:15.104710 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:40:15.104675 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b13014_57d9_4900_9916_50a147ff7f77.slice/crio-aa5ec2e07bf8093b658275bc4a876eb3cde6d3df1cb48b0ca11b35f870a1db4f WatchSource:0}: Error finding container aa5ec2e07bf8093b658275bc4a876eb3cde6d3df1cb48b0ca11b35f870a1db4f: Status 404 returned error can't find the container with id aa5ec2e07bf8093b658275bc4a876eb3cde6d3df1cb48b0ca11b35f870a1db4f Jul 15 22:40:15.243741 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:15.243653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" event={"ID":"c9b13014-57d9-4900-9916-50a147ff7f77","Type":"ContainerStarted","Data":"aa5ec2e07bf8093b658275bc4a876eb3cde6d3df1cb48b0ca11b35f870a1db4f"} Jul 15 22:40:15.319236 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:15.319201 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:16.787873 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:16.787835 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:17.145073 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.145038 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j"] Jul 15 22:40:17.151559 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.149376 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" Jul 15 22:40:17.153064 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.153040 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kvdj4\"" Jul 15 22:40:17.160965 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.160942 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j"] Jul 15 22:40:17.256146 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.256101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" event={"ID":"c9b13014-57d9-4900-9916-50a147ff7f77","Type":"ContainerStarted","Data":"e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3"} Jul 15 22:40:17.256352 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.256328 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:17.270625 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.270590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2hm\" (UniqueName: \"kubernetes.io/projected/be8d7941-2563-4761-ba43-d9835c34e994-kube-api-access-4w2hm\") pod \"authorino-operator-75cc8d5cb7-qhb4j\" (UID: \"be8d7941-2563-4761-ba43-d9835c34e994\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" Jul 15 22:40:17.283063 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.283015 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" podStartSLOduration=1.390778775 podStartE2EDuration="3.283000032s" totalImagesPullingTime="1.892221257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:40:17.282182067 +0000 UTC m=+428.222384609" watchObservedRunningTime="2026-07-15 22:40:17.283000032 +0000 UTC m=+428.223202583" Jul 15 22:40:17.371524 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.371433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2hm\" (UniqueName: \"kubernetes.io/projected/be8d7941-2563-4761-ba43-d9835c34e994-kube-api-access-4w2hm\") pod \"authorino-operator-75cc8d5cb7-qhb4j\" (UID: \"be8d7941-2563-4761-ba43-d9835c34e994\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" Jul 15 22:40:17.388919 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.388883 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2hm\" (UniqueName: \"kubernetes.io/projected/be8d7941-2563-4761-ba43-d9835c34e994-kube-api-access-4w2hm\") pod \"authorino-operator-75cc8d5cb7-qhb4j\" (UID: \"be8d7941-2563-4761-ba43-d9835c34e994\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" Jul 15 22:40:17.462242 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.462201 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" Jul 15 22:40:17.725141 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.725100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:17.825855 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:17.825816 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:18.354361 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:18.354290 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:18.489920 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:18.489898 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j"] Jul 15 22:40:18.492684 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:40:18.492657 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8d7941_2563_4761_ba43_d9835c34e994.slice/crio-61943712a2109b11846979967b54b8a0c9b8c5feb50f4f717e306963092938da WatchSource:0}: Error finding container 61943712a2109b11846979967b54b8a0c9b8c5feb50f4f717e306963092938da: Status 404 returned error can't find the container with id 61943712a2109b11846979967b54b8a0c9b8c5feb50f4f717e306963092938da Jul 15 22:40:19.252651 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:19.252618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:19.265389 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:19.265357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" event={"ID":"be8d7941-2563-4761-ba43-d9835c34e994","Type":"ContainerStarted","Data":"61943712a2109b11846979967b54b8a0c9b8c5feb50f4f717e306963092938da"} Jul 15 22:40:19.266809 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:19.266784 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" event={"ID":"d9423f4f-8eee-4573-9c89-a540028e97ce","Type":"ContainerStarted","Data":"c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d"} Jul 15 22:40:19.266949 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:19.266932 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:19.291237 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:19.291152 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" podStartSLOduration=1.996163944 podStartE2EDuration="7.291130887s" totalImagesPullingTime="5.294966943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:40:19.288957346 +0000 UTC m=+430.229159910" watchObservedRunningTime="2026-07-15 22:40:19.291130887 +0000 UTC m=+430.231333460" Jul 15 22:40:21.160365 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.160335 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:21.278196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.278097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" event={"ID":"be8d7941-2563-4761-ba43-d9835c34e994","Type":"ContainerStarted","Data":"cf778d5336acd89323e9d4d15e2ad27849094ecfbcf6b290613ef48e0a2932f4"} Jul 15 22:40:21.278351 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.278231 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" Jul 15 22:40:21.300592 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.298058 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" podStartSLOduration=1.874551364 podStartE2EDuration="4.298036768s" totalImagesPullingTime="2.423485404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:40:21.296308468 +0000 UTC m=+432.236511041" watchObservedRunningTime="2026-07-15 22:40:21.298036768 +0000 UTC m=+432.238239310" Jul 15 22:40:21.524431 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.524394 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl"] Jul 15 22:40:21.527732 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.527715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.530405 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.530342 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 22:40:21.530405 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.530351 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 22:40:21.530567 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.530424 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dm9m6\"" Jul 15 22:40:21.540621 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.540590 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:21.541254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.541233 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl"] Jul 15 22:40:21.684046 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.684008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:21.709221 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.709158 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln5pj\" (UniqueName: \"kubernetes.io/projected/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-kube-api-access-ln5pj\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.709381 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.709241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.709381 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.709273 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.810578 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.810475 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln5pj\" (UniqueName: \"kubernetes.io/projected/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-kube-api-access-ln5pj\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.810578 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.810523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.810578 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.810560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.811264 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.811241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.813033 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.812999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.821108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.821078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln5pj\" (UniqueName: \"kubernetes.io/projected/c246eba4-afd0-4cf9-b79f-4bf8415cb8fa-kube-api-access-ln5pj\") pod \"kuadrant-console-plugin-66b5774985-c7rxl\" (UID: \"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.837938 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.837917 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" Jul 15 22:40:21.972250 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:21.972089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl"] Jul 15 22:40:21.974653 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:40:21.974626 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc246eba4_afd0_4cf9_b79f_4bf8415cb8fa.slice/crio-03a35374ce30c866c34d2b3c701844e5f08a3a1bedc773aa859f2fbb6b1fa315 WatchSource:0}: Error finding container 03a35374ce30c866c34d2b3c701844e5f08a3a1bedc773aa859f2fbb6b1fa315: Status 404 returned error can't find the container with id 03a35374ce30c866c34d2b3c701844e5f08a3a1bedc773aa859f2fbb6b1fa315 Jul 15 22:40:22.283198 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:22.283148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" event={"ID":"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa","Type":"ContainerStarted","Data":"03a35374ce30c866c34d2b3c701844e5f08a3a1bedc773aa859f2fbb6b1fa315"} Jul 15 22:40:23.003699 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:23.003657 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:23.104648 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:23.104610 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:24.310665 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:24.310627 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:28.263511 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:28.263481 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:30.274547 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:30.274313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:30.578051 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:30.577968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:30.674724 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:30.674679 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:32.001122 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.001082 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd"] Jul 15 22:40:32.001633 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.001320 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" containerName="manager" containerID="cri-o://c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d" gracePeriod=2 Jul 15 22:40:32.013086 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.013059 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd"] Jul 15 22:40:32.020741 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.020697 2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d9423f4f-8eee-4573-9c89-a540028e97ce\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\": pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:32.029615 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.029587 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45"] Jul 15 22:40:32.029862 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.029824 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" containerName="manager" containerID="cri-o://e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3" gracePeriod=2 Jul 15 22:40:32.047224 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.047181 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45"] Jul 15 22:40:32.062216 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.061685 2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c9b13014-57d9-4900-9916-50a147ff7f77\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-pxx45\": pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:32.064510 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.064481 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg"] Jul 15 22:40:32.064663 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.064483 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:32.064945 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.064929 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" containerName="manager" Jul 15 22:40:32.065044 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.064948 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" containerName="manager" Jul 15 22:40:32.065044 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.064962 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" containerName="manager" Jul 15 22:40:32.065044 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.064969 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" containerName="manager" Jul 15 22:40:32.065044 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.065036 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" containerName="manager" Jul 15 22:40:32.065044 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.065045 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" containerName="manager" Jul 15 22:40:32.068519 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.068494 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.071672 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.071637 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:32.074402 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.074371 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:32.081748 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.081723 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk"] Jul 15 22:40:32.091890 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.091854 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhppg\" (UniqueName: \"kubernetes.io/projected/c7b20a34-802e-468a-8b32-4d82308bf972-kube-api-access-dhppg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whvtg\" (UID: \"c7b20a34-802e-468a-8b32-4d82308bf972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.091890 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.091885 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg"] Jul 15 22:40:32.092084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.091911 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7b20a34-802e-468a-8b32-4d82308bf972-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whvtg\" (UID: \"c7b20a34-802e-468a-8b32-4d82308bf972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.092084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.092002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" Jul 15 22:40:32.102971 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.102938 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk"] Jul 15 22:40:32.105053 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.105012 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:32.107801 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.107774 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:32.133731 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.133307 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg"] Jul 15 22:40:32.135077 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.135028 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-dhppg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" podUID="c7b20a34-802e-468a-8b32-4d82308bf972" Jul 15 22:40:32.164902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.164855 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8"] Jul 15 22:40:32.169794 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.169768 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:32.172472 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.172430 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:32.183450 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.183419 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8"] Jul 15 22:40:32.192952 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.192916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fpgh8\" (UID: \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:32.193129 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.193099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6jw\" (UniqueName: \"kubernetes.io/projected/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-kube-api-access-tn6jw\") pod \"kuadrant-operator-controller-manager-86878c769f-fpgh8\" (UID: \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:32.193289 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.193212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhppg\" (UniqueName: \"kubernetes.io/projected/c7b20a34-802e-468a-8b32-4d82308bf972-kube-api-access-dhppg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whvtg\" (UID: \"c7b20a34-802e-468a-8b32-4d82308bf972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.193289 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.193264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxb7\" (UniqueName: \"kubernetes.io/projected/a5228b7e-f25e-41f8-8fb8-8cb982becec2-kube-api-access-ltxb7\") pod \"limitador-operator-controller-manager-859d7b78bd-sz8bk\" (UID: \"a5228b7e-f25e-41f8-8fb8-8cb982becec2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" Jul 15 22:40:32.193400 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.193304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7b20a34-802e-468a-8b32-4d82308bf972-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whvtg\" (UID: \"c7b20a34-802e-468a-8b32-4d82308bf972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.193636 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.193604 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7b20a34-802e-468a-8b32-4d82308bf972-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whvtg\" (UID: \"c7b20a34-802e-468a-8b32-4d82308bf972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.208448 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.208391 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:32.211160 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.211133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhppg\" (UniqueName: \"kubernetes.io/projected/c7b20a34-802e-468a-8b32-4d82308bf972-kube-api-access-dhppg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whvtg\" (UID: \"c7b20a34-802e-468a-8b32-4d82308bf972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.286052 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.285966 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhb4j" Jul 15 22:40:32.293952 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.293915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fpgh8\" (UID: \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:32.294136 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.293969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6jw\" (UniqueName: \"kubernetes.io/projected/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-kube-api-access-tn6jw\") pod \"kuadrant-operator-controller-manager-86878c769f-fpgh8\" (UID: \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:32.294136 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.294004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxb7\" (UniqueName: \"kubernetes.io/projected/a5228b7e-f25e-41f8-8fb8-8cb982becec2-kube-api-access-ltxb7\") pod \"limitador-operator-controller-manager-859d7b78bd-sz8bk\" (UID: \"a5228b7e-f25e-41f8-8fb8-8cb982becec2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" Jul 15 22:40:32.294359 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.294330 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fpgh8\" (UID: \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:32.322420 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.322377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxb7\" (UniqueName: \"kubernetes.io/projected/a5228b7e-f25e-41f8-8fb8-8cb982becec2-kube-api-access-ltxb7\") pod \"limitador-operator-controller-manager-859d7b78bd-sz8bk\" (UID: \"a5228b7e-f25e-41f8-8fb8-8cb982becec2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" Jul 15 22:40:32.323310 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.323284 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6jw\" (UniqueName: \"kubernetes.io/projected/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-kube-api-access-tn6jw\") pod \"kuadrant-operator-controller-manager-86878c769f-fpgh8\" (UID: \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:32.323888 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.323854 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:32.330646 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.330605 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:32.334742 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.334411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.350648 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.350614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:32.360823 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.360781 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:32.363199 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.363151 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:32.366796 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.366762 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:32.368970 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:32.368942 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:32.394904 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.394862 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhppg\" (UniqueName: \"kubernetes.io/projected/c7b20a34-802e-468a-8b32-4d82308bf972-kube-api-access-dhppg\") pod \"c7b20a34-802e-468a-8b32-4d82308bf972\" (UID: \"c7b20a34-802e-468a-8b32-4d82308bf972\") " Jul 15 22:40:32.395079 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.394951 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7b20a34-802e-468a-8b32-4d82308bf972-extensions-socket-volume\") pod \"c7b20a34-802e-468a-8b32-4d82308bf972\" (UID: \"c7b20a34-802e-468a-8b32-4d82308bf972\") " Jul 15 22:40:32.395257 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.395217 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b20a34-802e-468a-8b32-4d82308bf972-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c7b20a34-802e-468a-8b32-4d82308bf972" (UID: "c7b20a34-802e-468a-8b32-4d82308bf972"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:40:32.397471 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.397442 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b20a34-802e-468a-8b32-4d82308bf972-kube-api-access-dhppg" (OuterVolumeSpecName: "kube-api-access-dhppg") pod "c7b20a34-802e-468a-8b32-4d82308bf972" (UID: "c7b20a34-802e-468a-8b32-4d82308bf972"). InnerVolumeSpecName "kube-api-access-dhppg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:40:32.444762 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.444723 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" Jul 15 22:40:32.482600 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.482565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:32.495663 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.495635 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7b20a34-802e-468a-8b32-4d82308bf972-extensions-socket-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:40:32.495663 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:32.495663 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhppg\" (UniqueName: \"kubernetes.io/projected/c7b20a34-802e-468a-8b32-4d82308bf972-kube-api-access-dhppg\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:40:33.339526 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:33.339491 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg" Jul 15 22:40:33.347480 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:33.347446 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:33.349602 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:33.349425 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:33.383677 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:33.383643 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg"] Jul 15 22:40:33.387723 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:33.387682 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:33.388513 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:33.388487 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whvtg"] Jul 15 22:40:33.389705 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:33.389674 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:33.585541 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:33.585507 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b20a34-802e-468a-8b32-4d82308bf972" path="/var/lib/kubelet/pods/c7b20a34-802e-468a-8b32-4d82308bf972/volumes" Jul 15 22:40:35.445058 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:35.445018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:35.528407 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:35.528372 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:37.826285 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:37.826237 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:37.986332 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:37.986298 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:43.351944 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:43.351905 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:43.452368 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:43.452332 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:43.987949 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:43.987919 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:43.990364 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:43.990334 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:43.994844 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:43.994823 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:43.996205 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:43.996159 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk"] Jul 15 22:40:43.997239 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:43.997161 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:43.998038 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:40:43.997998 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5228b7e_f25e_41f8_8fb8_8cb982becec2.slice/crio-dd10a05707fdef2aaf6fbd2ccc2d0c6ff2a31487f3536763d1a0bca70d583b0f WatchSource:0}: Error finding container dd10a05707fdef2aaf6fbd2ccc2d0c6ff2a31487f3536763d1a0bca70d583b0f: Status 404 returned error can't find the container with id dd10a05707fdef2aaf6fbd2ccc2d0c6ff2a31487f3536763d1a0bca70d583b0f Jul 15 22:40:43.999318 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:43.999284 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:44.000209 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.000102 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5grf\" (UniqueName: \"kubernetes.io/projected/c9b13014-57d9-4900-9916-50a147ff7f77-kube-api-access-p5grf\") pod \"c9b13014-57d9-4900-9916-50a147ff7f77\" (UID: \"c9b13014-57d9-4900-9916-50a147ff7f77\") " Jul 15 22:40:44.009862 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.009830 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b13014-57d9-4900-9916-50a147ff7f77-kube-api-access-p5grf" (OuterVolumeSpecName: "kube-api-access-p5grf") pod "c9b13014-57d9-4900-9916-50a147ff7f77" (UID: "c9b13014-57d9-4900-9916-50a147ff7f77"). InnerVolumeSpecName "kube-api-access-p5grf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:40:44.017660 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.017634 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8"] Jul 15 22:40:44.020366 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:40:44.020338 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c9f460_21c2_4ef6_913f_7c0e2734d0e4.slice/crio-a59454f673b044ac5dd581142539e051d6f17c4b39b76249dc839af4b38faec2 WatchSource:0}: Error finding container a59454f673b044ac5dd581142539e051d6f17c4b39b76249dc839af4b38faec2: Status 404 returned error can't find the container with id a59454f673b044ac5dd581142539e051d6f17c4b39b76249dc839af4b38faec2 Jul 15 22:40:44.101426 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.101395 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9423f4f-8eee-4573-9c89-a540028e97ce-extensions-socket-volume\") pod \"d9423f4f-8eee-4573-9c89-a540028e97ce\" (UID: \"d9423f4f-8eee-4573-9c89-a540028e97ce\") " Jul 15 22:40:44.101606 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.101461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg4pr\" (UniqueName: \"kubernetes.io/projected/d9423f4f-8eee-4573-9c89-a540028e97ce-kube-api-access-kg4pr\") pod \"d9423f4f-8eee-4573-9c89-a540028e97ce\" (UID: \"d9423f4f-8eee-4573-9c89-a540028e97ce\") " Jul 15 22:40:44.101673 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.101655 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5grf\" (UniqueName: \"kubernetes.io/projected/c9b13014-57d9-4900-9916-50a147ff7f77-kube-api-access-p5grf\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:40:44.102084 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.102050 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9423f4f-8eee-4573-9c89-a540028e97ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d9423f4f-8eee-4573-9c89-a540028e97ce" (UID: "d9423f4f-8eee-4573-9c89-a540028e97ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:40:44.103851 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.103827 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9423f4f-8eee-4573-9c89-a540028e97ce-kube-api-access-kg4pr" (OuterVolumeSpecName: "kube-api-access-kg4pr") pod "d9423f4f-8eee-4573-9c89-a540028e97ce" (UID: "d9423f4f-8eee-4573-9c89-a540028e97ce"). InnerVolumeSpecName "kube-api-access-kg4pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:40:44.202468 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.202425 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9423f4f-8eee-4573-9c89-a540028e97ce-extensions-socket-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:40:44.202468 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.202464 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg4pr\" (UniqueName: \"kubernetes.io/projected/d9423f4f-8eee-4573-9c89-a540028e97ce-kube-api-access-kg4pr\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:40:44.394828 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.394771 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" event={"ID":"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4","Type":"ContainerStarted","Data":"de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019"} Jul 15 22:40:44.395340 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.394936 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:44.395340 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.394983 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" event={"ID":"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4","Type":"ContainerStarted","Data":"a59454f673b044ac5dd581142539e051d6f17c4b39b76249dc839af4b38faec2"} Jul 15 22:40:44.396592 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.396545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" event={"ID":"c246eba4-afd0-4cf9-b79f-4bf8415cb8fa","Type":"ContainerStarted","Data":"55a793e729e789886d18364abb81baf0ac071bcd305cfbe5a7215bcff8921727"} Jul 15 22:40:44.398145 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.398118 2565 generic.go:356] "Generic (PLEG): container finished" podID="c9b13014-57d9-4900-9916-50a147ff7f77" containerID="e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3" exitCode=0 Jul 15 22:40:44.398288 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.398213 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:44.398288 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.398225 2565 scope.go:117] "RemoveContainer" containerID="e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3" Jul 15 22:40:44.400898 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.400813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" event={"ID":"a5228b7e-f25e-41f8-8fb8-8cb982becec2","Type":"ContainerStarted","Data":"5fa4c0340d6c6a770e05eef737979f4602f3cef8b209e3d303d8b2f9da20f33b"} Jul 15 22:40:44.400898 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.400859 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" event={"ID":"a5228b7e-f25e-41f8-8fb8-8cb982becec2","Type":"ContainerStarted","Data":"dd10a05707fdef2aaf6fbd2ccc2d0c6ff2a31487f3536763d1a0bca70d583b0f"} Jul 15 22:40:44.401184 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.401121 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" Jul 15 22:40:44.402705 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.402681 2565 generic.go:356] "Generic (PLEG): container finished" podID="d9423f4f-8eee-4573-9c89-a540028e97ce" containerID="c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d" exitCode=0 Jul 15 22:40:44.402797 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.402730 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:44.407740 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.407644 2565 scope.go:117] "RemoveContainer" containerID="e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3" Jul 15 22:40:44.408089 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:44.408060 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3\": container with ID starting with e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3 not found: ID does not exist" containerID="e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3" Jul 15 22:40:44.408198 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.408097 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3"} err="failed to get container status \"e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3\": rpc error: code = NotFound desc = could not find container \"e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3\": container with ID starting with e7dac11ed304af1254548b4bd050b7e8fee3a9bb0711d93737987b2b22b227c3 not found: ID does not exist" Jul 15 22:40:44.408198 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.408120 2565 scope.go:117] "RemoveContainer" containerID="c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d" Jul 15 22:40:44.415894 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.415876 2565 scope.go:117] "RemoveContainer" containerID="c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d" Jul 15 22:40:44.416273 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:44.416226 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d\": container with ID starting with c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d not found: ID does not exist" containerID="c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d" Jul 15 22:40:44.416273 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.416259 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d"} err="failed to get container status \"c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d\": rpc error: code = NotFound desc = could not find container \"c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d\": container with ID starting with c3f8364c59c2692610ed7df606184b5c005e3d7298b56776547db1b3a11b276d not found: ID does not exist" Jul 15 22:40:44.417525 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.417489 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" podStartSLOduration=12.417478117 podStartE2EDuration="12.417478117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:40:44.416281744 +0000 UTC m=+455.356484294" watchObservedRunningTime="2026-07-15 22:40:44.417478117 +0000 UTC m=+455.357680666" Jul 15 22:40:44.418588 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:44.418566 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:44.420644 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:44.420624 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:44.422629 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:44.422609 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pxx45\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pxx45" Jul 15 22:40:44.439902 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.439857 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c7rxl" podStartSLOduration=1.519641588 podStartE2EDuration="23.439842848s" totalImagesPullingTime="21.92020126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:40:44.438674749 +0000 UTC m=+455.378877300" watchObservedRunningTime="2026-07-15 22:40:44.439842848 +0000 UTC m=+455.380045398" Jul 15 22:40:44.441059 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:40:44.441036 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7p7wd\" is forbidden: User \"system:node:ip-10-0-130-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-194.ec2.internal' and this object" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7p7wd" Jul 15 22:40:44.459719 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:44.459664 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" podStartSLOduration=12.459644917 podStartE2EDuration="12.459644917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:40:44.458030283 +0000 UTC m=+455.398232833" watchObservedRunningTime="2026-07-15 22:40:44.459644917 +0000 UTC m=+455.399847468" Jul 15 22:40:45.586313 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:45.586273 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b13014-57d9-4900-9916-50a147ff7f77" path="/var/lib/kubelet/pods/c9b13014-57d9-4900-9916-50a147ff7f77/volumes" Jul 15 22:40:45.586791 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:45.586629 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9423f4f-8eee-4573-9c89-a540028e97ce" path="/var/lib/kubelet/pods/d9423f4f-8eee-4573-9c89-a540028e97ce/volumes" Jul 15 22:40:50.754702 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:50.754663 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:50.861076 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:50.861018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:55.411380 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:55.411349 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:40:55.411841 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:55.411423 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sz8bk" Jul 15 22:40:56.370925 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:56.370894 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:56.470836 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:56.470803 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:56.953345 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:56.953309 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:40:57.053420 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:40:57.053368 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:00.749367 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:00.749324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:00.849641 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:00.849608 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:06.775711 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:06.775649 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:06.875476 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:06.875434 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:11.810881 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:11.810805 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:12.631666 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:12.631617 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:13.953323 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:13.953276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:14.059489 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.059450 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:14.354729 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.354637 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:14.445903 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.445861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:14.503989 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.503954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:14.671641 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.671604 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:41:14.733273 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.733235 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:41:14.733273 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.733277 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:41:14.733486 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.733350 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:14.735869 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.735843 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 22:41:14.869341 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.869307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296l6\" (UniqueName: \"kubernetes.io/projected/d2195f4a-fa10-48fa-9813-bd40bf370ab4-kube-api-access-296l6\") pod \"limitador-limitador-54d6d988d5-brmnx\" (UID: \"d2195f4a-fa10-48fa-9813-bd40bf370ab4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:14.869520 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.869347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2195f4a-fa10-48fa-9813-bd40bf370ab4-config-file\") pod \"limitador-limitador-54d6d988d5-brmnx\" (UID: \"d2195f4a-fa10-48fa-9813-bd40bf370ab4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:14.970749 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.970665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296l6\" (UniqueName: \"kubernetes.io/projected/d2195f4a-fa10-48fa-9813-bd40bf370ab4-kube-api-access-296l6\") pod \"limitador-limitador-54d6d988d5-brmnx\" (UID: \"d2195f4a-fa10-48fa-9813-bd40bf370ab4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:14.970749 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.970710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2195f4a-fa10-48fa-9813-bd40bf370ab4-config-file\") pod \"limitador-limitador-54d6d988d5-brmnx\" (UID: \"d2195f4a-fa10-48fa-9813-bd40bf370ab4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:14.971462 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.971443 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2195f4a-fa10-48fa-9813-bd40bf370ab4-config-file\") pod \"limitador-limitador-54d6d988d5-brmnx\" (UID: \"d2195f4a-fa10-48fa-9813-bd40bf370ab4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:14.979268 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:14.979243 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-296l6\" (UniqueName: \"kubernetes.io/projected/d2195f4a-fa10-48fa-9813-bd40bf370ab4-kube-api-access-296l6\") pod \"limitador-limitador-54d6d988d5-brmnx\" (UID: \"d2195f4a-fa10-48fa-9813-bd40bf370ab4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:15.043230 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.043184 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:15.132607 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.132530 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-p42f4"] Jul 15 22:41:15.136424 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.136403 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p42f4" Jul 15 22:41:15.140748 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.140720 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7ghtb\"" Jul 15 22:41:15.146478 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.146456 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-p42f4"] Jul 15 22:41:15.165962 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.165932 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:15.189734 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.189710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:41:15.192339 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:41:15.192313 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2195f4a_fa10_48fa_9813_bd40bf370ab4.slice/crio-261fdf945c90aa5395631a2f23373436840953e19b22f69cff9ec5555c5c1479 WatchSource:0}: Error finding container 261fdf945c90aa5395631a2f23373436840953e19b22f69cff9ec5555c5c1479: Status 404 returned error can't find the container with id 261fdf945c90aa5395631a2f23373436840953e19b22f69cff9ec5555c5c1479 Jul 15 22:41:15.266466 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.266379 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:15.273514 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.273484 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdln\" (UniqueName: \"kubernetes.io/projected/32493e1b-780f-45ec-be95-b4bbfb76b5ca-kube-api-access-zmdln\") pod \"authorino-58cd955675-p42f4\" (UID: \"32493e1b-780f-45ec-be95-b4bbfb76b5ca\") " pod="kuadrant-system/authorino-58cd955675-p42f4" Jul 15 22:41:15.374307 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.374270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdln\" (UniqueName: \"kubernetes.io/projected/32493e1b-780f-45ec-be95-b4bbfb76b5ca-kube-api-access-zmdln\") pod \"authorino-58cd955675-p42f4\" (UID: \"32493e1b-780f-45ec-be95-b4bbfb76b5ca\") " pod="kuadrant-system/authorino-58cd955675-p42f4" Jul 15 22:41:15.382373 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.382342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdln\" (UniqueName: \"kubernetes.io/projected/32493e1b-780f-45ec-be95-b4bbfb76b5ca-kube-api-access-zmdln\") pod \"authorino-58cd955675-p42f4\" (UID: \"32493e1b-780f-45ec-be95-b4bbfb76b5ca\") " pod="kuadrant-system/authorino-58cd955675-p42f4" Jul 15 22:41:15.449782 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.449748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p42f4" Jul 15 22:41:15.542597 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.542562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" event={"ID":"d2195f4a-fa10-48fa-9813-bd40bf370ab4","Type":"ContainerStarted","Data":"261fdf945c90aa5395631a2f23373436840953e19b22f69cff9ec5555c5c1479"} Jul 15 22:41:15.585710 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:15.585687 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-p42f4"] Jul 15 22:41:15.586784 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:41:15.586757 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32493e1b_780f_45ec_be95_b4bbfb76b5ca.slice/crio-ef4cfa88de7de916fe00f6154c59fd3c065d0f51f4795404c8deba9320ac84e1 WatchSource:0}: Error finding container ef4cfa88de7de916fe00f6154c59fd3c065d0f51f4795404c8deba9320ac84e1: Status 404 returned error can't find the container with id ef4cfa88de7de916fe00f6154c59fd3c065d0f51f4795404c8deba9320ac84e1 Jul 15 22:41:16.550965 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:16.550922 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p42f4" event={"ID":"32493e1b-780f-45ec-be95-b4bbfb76b5ca","Type":"ContainerStarted","Data":"ef4cfa88de7de916fe00f6154c59fd3c065d0f51f4795404c8deba9320ac84e1"} Jul 15 22:41:18.561738 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:18.561542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:18.664704 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:18.664670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:19.936760 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:19.936724 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-p42f4"] Jul 15 22:41:20.572559 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:20.572519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p42f4" event={"ID":"32493e1b-780f-45ec-be95-b4bbfb76b5ca","Type":"ContainerStarted","Data":"2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c"} Jul 15 22:41:20.574012 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:20.573983 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" event={"ID":"d2195f4a-fa10-48fa-9813-bd40bf370ab4","Type":"ContainerStarted","Data":"17d339b2507533cdae06c857af20b909d37412c0d0b23041082974aba4dbcbc3"} Jul 15 22:41:20.574190 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:20.574132 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:20.587938 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:20.587893 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-p42f4" podStartSLOduration=1.520637094 podStartE2EDuration="5.587879457s" totalImagesPullingTime="4.067242363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:41:20.587115221 +0000 UTC m=+491.527317772" watchObservedRunningTime="2026-07-15 22:41:20.587879457 +0000 UTC m=+491.528082007" Jul 15 22:41:20.605367 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:20.605317 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" podStartSLOduration=2.195061472 podStartE2EDuration="6.605302367s" totalImagesPullingTime="4.410240895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:41:20.603980299 +0000 UTC m=+491.544182849" watchObservedRunningTime="2026-07-15 22:41:20.605302367 +0000 UTC m=+491.545505307" Jul 15 22:41:21.012576 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:21.012518 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:21.113118 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:21.113076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:21.578253 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:21.578212 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-p42f4" podUID="32493e1b-780f-45ec-be95-b4bbfb76b5ca" containerName="authorino" containerID="cri-o://2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c" gracePeriod=30 Jul 15 22:41:21.833798 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:21.833739 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p42f4" Jul 15 22:41:21.932047 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:21.932017 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmdln\" (UniqueName: \"kubernetes.io/projected/32493e1b-780f-45ec-be95-b4bbfb76b5ca-kube-api-access-zmdln\") pod \"32493e1b-780f-45ec-be95-b4bbfb76b5ca\" (UID: \"32493e1b-780f-45ec-be95-b4bbfb76b5ca\") " Jul 15 22:41:21.934202 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:21.934158 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32493e1b-780f-45ec-be95-b4bbfb76b5ca-kube-api-access-zmdln" (OuterVolumeSpecName: "kube-api-access-zmdln") pod "32493e1b-780f-45ec-be95-b4bbfb76b5ca" (UID: "32493e1b-780f-45ec-be95-b4bbfb76b5ca"). InnerVolumeSpecName "kube-api-access-zmdln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:41:22.032716 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.032677 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmdln\" (UniqueName: \"kubernetes.io/projected/32493e1b-780f-45ec-be95-b4bbfb76b5ca-kube-api-access-zmdln\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:41:22.584553 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.584519 2565 generic.go:356] "Generic (PLEG): container finished" podID="32493e1b-780f-45ec-be95-b4bbfb76b5ca" containerID="2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c" exitCode=0 Jul 15 22:41:22.584553 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.584553 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p42f4" Jul 15 22:41:22.584800 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.584597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p42f4" event={"ID":"32493e1b-780f-45ec-be95-b4bbfb76b5ca","Type":"ContainerDied","Data":"2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c"} Jul 15 22:41:22.584800 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.584627 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p42f4" event={"ID":"32493e1b-780f-45ec-be95-b4bbfb76b5ca","Type":"ContainerDied","Data":"ef4cfa88de7de916fe00f6154c59fd3c065d0f51f4795404c8deba9320ac84e1"} Jul 15 22:41:22.584800 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.584644 2565 scope.go:117] "RemoveContainer" containerID="2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c" Jul 15 22:41:22.593578 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.593558 2565 scope.go:117] "RemoveContainer" containerID="2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c" Jul 15 22:41:22.593831 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:41:22.593810 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c\": container with ID starting with 2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c not found: ID does not exist" containerID="2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c" Jul 15 22:41:22.593909 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.593843 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c"} err="failed to get container status \"2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c\": rpc error: code = NotFound desc = could not find container \"2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c\": container with ID starting with 2ecfbc1ede402d651a75e7ad675a0f2028269603f349ab8476bfa16c9838ae2c not found: ID does not exist" Jul 15 22:41:22.605200 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.605156 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-p42f4"] Jul 15 22:41:22.608772 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:22.608750 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-p42f4"] Jul 15 22:41:23.585410 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:23.585378 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32493e1b-780f-45ec-be95-b4bbfb76b5ca" path="/var/lib/kubelet/pods/32493e1b-780f-45ec-be95-b4bbfb76b5ca/volumes" Jul 15 22:41:31.584006 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:31.583965 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-brmnx" Jul 15 22:41:35.399244 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:35.399209 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:35.498595 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:35.498548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:35.920708 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:35.920676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:48.402108 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:48.402071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:48.502805 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:48.502770 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:54.694501 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:54.694463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:54.759956 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:54.759912 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:54.825297 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:54.825257 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:54.897508 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:54.897469 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:58.207299 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:58.207263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:41:58.292138 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:41:58.292101 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:28.843537 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:28.843499 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:28.940136 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:28.940100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:29.336776 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:29.336737 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:29.457682 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:29.457639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:29.992939 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:29.992905 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:30.100861 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:30.100826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:30.170074 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:30.170044 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:42:35.928479 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:35.928444 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:39.238897 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:39.238848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:39.388039 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:39.388002 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:39.489612 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:39.489504 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:55.903363 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:55.903323 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:42:56.450631 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:42:56.450590 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:43:09.392867 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:09.392825 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:43:10.247212 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:10.247159 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:43:17.606967 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:17.606922 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:43:18.665277 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:18.665236 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:43:32.258502 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.258408 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk"] Jul 15 22:43:32.259067 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.258932 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32493e1b-780f-45ec-be95-b4bbfb76b5ca" containerName="authorino" Jul 15 22:43:32.259067 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.258952 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32493e1b-780f-45ec-be95-b4bbfb76b5ca" containerName="authorino" Jul 15 22:43:32.259067 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.259044 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="32493e1b-780f-45ec-be95-b4bbfb76b5ca" containerName="authorino" Jul 15 22:43:32.261981 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.261959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.267564 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.267536 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 15 22:43:32.267715 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.267572 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 22:43:32.267715 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.267655 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d7k42\"" Jul 15 22:43:32.267715 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.267545 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 22:43:32.272792 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.272771 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk"] Jul 15 22:43:32.337120 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.337087 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.337120 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.337131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844v7\" (UniqueName: \"kubernetes.io/projected/bf74c3eb-eacb-4435-9b36-dea6921b5910-kube-api-access-844v7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.337371 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.337209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.337371 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.337257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.337371 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.337290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf74c3eb-eacb-4435-9b36-dea6921b5910-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.337371 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.337324 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.355691 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.355660 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:43:32.438443 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.438411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.438650 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.438463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844v7\" (UniqueName: \"kubernetes.io/projected/bf74c3eb-eacb-4435-9b36-dea6921b5910-kube-api-access-844v7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.438650 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.438513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.438650 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.438547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.438650 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.438572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf74c3eb-eacb-4435-9b36-dea6921b5910-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.438650 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.438619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.438935 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.438910 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.438975 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.438957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.439048 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.439032 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.441022 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.441001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf74c3eb-eacb-4435-9b36-dea6921b5910-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.441257 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.441239 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf74c3eb-eacb-4435-9b36-dea6921b5910-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.447555 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.447533 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-844v7\" (UniqueName: \"kubernetes.io/projected/bf74c3eb-eacb-4435-9b36-dea6921b5910-kube-api-access-844v7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk\" (UID: \"bf74c3eb-eacb-4435-9b36-dea6921b5910\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.572706 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.572618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:32.709884 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.709857 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk"] Jul 15 22:43:32.711985 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:43:32.711954 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf74c3eb_eacb_4435_9b36_dea6921b5910.slice/crio-b14f48b9b563069b25b404470c27a896e92c7f0d644e8a1b83855d26f6a2ef16 WatchSource:0}: Error finding container b14f48b9b563069b25b404470c27a896e92c7f0d644e8a1b83855d26f6a2ef16: Status 404 returned error can't find the container with id b14f48b9b563069b25b404470c27a896e92c7f0d644e8a1b83855d26f6a2ef16 Jul 15 22:43:32.713474 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:32.713448 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 22:43:33.151990 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:33.151911 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" event={"ID":"bf74c3eb-eacb-4435-9b36-dea6921b5910","Type":"ContainerStarted","Data":"b14f48b9b563069b25b404470c27a896e92c7f0d644e8a1b83855d26f6a2ef16"} Jul 15 22:43:34.158338 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:34.158306 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" event={"ID":"bf74c3eb-eacb-4435-9b36-dea6921b5910","Type":"ContainerStarted","Data":"3fe4f20389258df364f524677c2eb76b1725e4084681e7902a396b16f5794baa"} Jul 15 22:43:34.175785 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:34.175726 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" podStartSLOduration=1.166532573 podStartE2EDuration="2.175709491s" totalImagesPullingTime="1.009176918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:43:34.174970988 +0000 UTC m=+625.115173539" watchObservedRunningTime="2026-07-15 22:43:34.175709491 +0000 UTC m=+625.115912099" Jul 15 22:43:34.357723 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:34.357688 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:43:42.573447 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:42.573403 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:42.573875 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:42.573491 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:42.586133 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:42.586107 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:43.213264 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:43.213235 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk" Jul 15 22:43:47.995004 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:47.994972 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:43:50.259260 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:43:50.259226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:44:01.778276 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.778240 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf"] Jul 15 22:44:01.781368 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.781349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.784692 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.784671 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 22:44:01.801984 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.801952 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf"] Jul 15 22:44:01.857440 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.857406 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:44:01.894569 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.894529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.894759 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.894597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.894759 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.894675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.894759 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.894738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.894908 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.894765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkkt\" (UniqueName: \"kubernetes.io/projected/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-kube-api-access-7nkkt\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.894908 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.894808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.995883 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.995845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.996073 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.995899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.996073 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.995936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.996073 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.995963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.996073 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.995984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.996073 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.996008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkkt\" (UniqueName: \"kubernetes.io/projected/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-kube-api-access-7nkkt\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.996444 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.996420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.996516 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.996479 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.996516 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.996497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.998385 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.998357 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:01.998627 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:01.998609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:02.007771 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:02.007740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkkt\" (UniqueName: \"kubernetes.io/projected/a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc-kube-api-access-7nkkt\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf\" (UID: \"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:02.090305 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:02.090198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:02.243713 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:02.243678 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf"] Jul 15 22:44:02.246664 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:44:02.246633 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda02c8d76_8d18_4ff6_82d1_9a2bdf1457bc.slice/crio-bbde2ff4c1e0a0eb8c10958102818a11f17095cb7dae225bb2a5f8ce0a8ec782 WatchSource:0}: Error finding container bbde2ff4c1e0a0eb8c10958102818a11f17095cb7dae225bb2a5f8ce0a8ec782: Status 404 returned error can't find the container with id bbde2ff4c1e0a0eb8c10958102818a11f17095cb7dae225bb2a5f8ce0a8ec782 Jul 15 22:44:02.285960 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:02.285924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" event={"ID":"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc","Type":"ContainerStarted","Data":"bbde2ff4c1e0a0eb8c10958102818a11f17095cb7dae225bb2a5f8ce0a8ec782"} Jul 15 22:44:03.292071 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:03.292031 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" event={"ID":"a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc","Type":"ContainerStarted","Data":"0222f0317326fbcb7bd5289e35f857594b292da6b23099a7dded7303846f1c8e"} Jul 15 22:44:03.310522 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:03.310463 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" podStartSLOduration=2.124477444 podStartE2EDuration="2.31044467s" totalImagesPullingTime="185.967226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:44:03.3093655 +0000 UTC m=+654.249568139" watchObservedRunningTime="2026-07-15 22:44:03.31044467 +0000 UTC m=+654.250647220" Jul 15 22:44:04.572718 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:04.572680 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:44:12.090375 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:12.090335 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:12.090871 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:12.090399 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:12.110729 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:12.110693 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:12.351987 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:12.351904 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf" Jul 15 22:44:36.849124 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:36.849089 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:44:36.950294 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:36.950256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:44:57.466327 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:57.466296 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:44:57.519595 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:44:57.519557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:45:25.854728 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.854644 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56b9548c98-sm2cz"] Jul 15 22:45:25.858116 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.858096 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b9548c98-sm2cz" Jul 15 22:45:25.860922 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.860902 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7ghtb\"" Jul 15 22:45:25.862052 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.862036 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 22:45:25.865931 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.865911 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56b9548c98-sm2cz"] Jul 15 22:45:25.896407 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.896372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ebe998d-d511-4e83-97a1-0bb900356251-tls-cert\") pod \"authorino-56b9548c98-sm2cz\" (UID: \"6ebe998d-d511-4e83-97a1-0bb900356251\") " pod="kuadrant-system/authorino-56b9548c98-sm2cz" Jul 15 22:45:25.896554 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.896429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828hg\" (UniqueName: \"kubernetes.io/projected/6ebe998d-d511-4e83-97a1-0bb900356251-kube-api-access-828hg\") pod \"authorino-56b9548c98-sm2cz\" (UID: \"6ebe998d-d511-4e83-97a1-0bb900356251\") " pod="kuadrant-system/authorino-56b9548c98-sm2cz" Jul 15 22:45:25.997780 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.997739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ebe998d-d511-4e83-97a1-0bb900356251-tls-cert\") pod \"authorino-56b9548c98-sm2cz\" (UID: \"6ebe998d-d511-4e83-97a1-0bb900356251\") " pod="kuadrant-system/authorino-56b9548c98-sm2cz" Jul 15 22:45:25.997983 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:25.997813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828hg\" (UniqueName: \"kubernetes.io/projected/6ebe998d-d511-4e83-97a1-0bb900356251-kube-api-access-828hg\") pod \"authorino-56b9548c98-sm2cz\" (UID: \"6ebe998d-d511-4e83-97a1-0bb900356251\") " pod="kuadrant-system/authorino-56b9548c98-sm2cz" Jul 15 22:45:26.000482 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:26.000452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ebe998d-d511-4e83-97a1-0bb900356251-tls-cert\") pod \"authorino-56b9548c98-sm2cz\" (UID: \"6ebe998d-d511-4e83-97a1-0bb900356251\") " pod="kuadrant-system/authorino-56b9548c98-sm2cz" Jul 15 22:45:26.005996 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:26.005973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-828hg\" (UniqueName: \"kubernetes.io/projected/6ebe998d-d511-4e83-97a1-0bb900356251-kube-api-access-828hg\") pod \"authorino-56b9548c98-sm2cz\" (UID: \"6ebe998d-d511-4e83-97a1-0bb900356251\") " pod="kuadrant-system/authorino-56b9548c98-sm2cz" Jul 15 22:45:26.168144 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:26.168095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b9548c98-sm2cz" Jul 15 22:45:26.300038 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:26.300000 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56b9548c98-sm2cz"] Jul 15 22:45:26.303848 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:45:26.303807 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ebe998d_d511_4e83_97a1_0bb900356251.slice/crio-cfeb4143258fbb3d04832cc8a6a388b8c2df5263fdfac7a76ca1b0af419b889e WatchSource:0}: Error finding container cfeb4143258fbb3d04832cc8a6a388b8c2df5263fdfac7a76ca1b0af419b889e: Status 404 returned error can't find the container with id cfeb4143258fbb3d04832cc8a6a388b8c2df5263fdfac7a76ca1b0af419b889e Jul 15 22:45:26.674650 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:26.674618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b9548c98-sm2cz" event={"ID":"6ebe998d-d511-4e83-97a1-0bb900356251","Type":"ContainerStarted","Data":"cfeb4143258fbb3d04832cc8a6a388b8c2df5263fdfac7a76ca1b0af419b889e"} Jul 15 22:45:27.681552 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:27.681513 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b9548c98-sm2cz" event={"ID":"6ebe998d-d511-4e83-97a1-0bb900356251","Type":"ContainerStarted","Data":"16b534a40c023a6a092eb889c7bf5371d2aa99b0b30c258763ff7d7d612ef3c2"} Jul 15 22:45:27.697981 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:27.697914 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56b9548c98-sm2cz" podStartSLOduration=2.149804985 podStartE2EDuration="2.697892483s" totalImagesPullingTime="548.087498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:45:27.696885061 +0000 UTC m=+738.637087613" watchObservedRunningTime="2026-07-15 22:45:27.697892483 +0000 UTC m=+738.638095034" Jul 15 22:45:28.223599 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:28.223558 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:45:28.323656 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:28.323613 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:45:29.377439 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:29.377396 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:45:29.500391 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:29.500356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:45:47.817570 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:47.817528 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:45:47.918395 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:47.918358 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:45:56.747861 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:45:56.747822 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:46:00.283626 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:46:00.283598 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:46:06.352772 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:46:06.352723 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:46:16.847940 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:46:16.847893 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:46:17.797139 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:46:17.797106 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:46:24.676779 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:46:24.676738 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:47:32.349676 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:47:32.349640 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:47:33.566496 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:47:33.566460 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:48:00.349232 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:00.349142 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:48:07.739319 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:07.739285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:48:07.837931 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:07.837896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:48:07.947147 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:07.947101 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:48:09.646522 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:09.646483 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:48:13.625990 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:13.625952 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:48:15.013405 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:15.013367 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:48:19.832256 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:19.832214 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:48:19.896048 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:19.896011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:48:37.043728 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:37.043693 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:48:46.649913 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:46.649873 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:48:56.946766 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:48:56.946726 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:49:02.847002 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:49:02.846960 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:49:14.167196 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:49:14.164023 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:49:24.158659 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:49:24.158620 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:49:34.954232 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:49:34.954136 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:49:45.243443 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:49:45.243406 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:49:46.081990 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:49:46.081946 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:50:06.346789 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:50:06.346748 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:50:17.353152 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:50:17.353110 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:50:41.063322 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:50:41.063289 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:50:51.857448 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:50:51.857407 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:50:52.446878 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:50:52.446826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:51:08.256916 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:51:08.256825 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:51:17.859900 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:51:17.859863 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:51:27.554935 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:51:27.554878 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:51:39.656203 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:51:39.656152 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:52:10.669858 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:10.669819 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8"] Jul 15 22:52:10.670360 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:10.670001 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" podUID="e6c9f460-21c2-4ef6-913f-7c0e2734d0e4" containerName="manager" containerID="cri-o://de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019" gracePeriod=10 Jul 15 22:52:13.028456 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.028431 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:52:13.079759 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.079667 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn6jw\" (UniqueName: \"kubernetes.io/projected/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-kube-api-access-tn6jw\") pod \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\" (UID: \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\") " Jul 15 22:52:13.079759 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.079734 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-extensions-socket-volume\") pod \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\" (UID: \"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4\") " Jul 15 22:52:13.080147 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.080122 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e6c9f460-21c2-4ef6-913f-7c0e2734d0e4" (UID: "e6c9f460-21c2-4ef6-913f-7c0e2734d0e4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:52:13.081867 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.081830 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-kube-api-access-tn6jw" (OuterVolumeSpecName: "kube-api-access-tn6jw") pod "e6c9f460-21c2-4ef6-913f-7c0e2734d0e4" (UID: "e6c9f460-21c2-4ef6-913f-7c0e2734d0e4"). InnerVolumeSpecName "kube-api-access-tn6jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:52:13.180804 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.180766 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6jw\" (UniqueName: \"kubernetes.io/projected/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-kube-api-access-tn6jw\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:52:13.180804 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.180797 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4-extensions-socket-volume\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 22:52:13.538493 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.538450 2565 generic.go:356] "Generic (PLEG): container finished" podID="e6c9f460-21c2-4ef6-913f-7c0e2734d0e4" containerID="de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019" exitCode=0 Jul 15 22:52:13.538694 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.538497 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" Jul 15 22:52:13.538694 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.538509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" event={"ID":"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4","Type":"ContainerDied","Data":"de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019"} Jul 15 22:52:13.538694 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.538605 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8" event={"ID":"e6c9f460-21c2-4ef6-913f-7c0e2734d0e4","Type":"ContainerDied","Data":"a59454f673b044ac5dd581142539e051d6f17c4b39b76249dc839af4b38faec2"} Jul 15 22:52:13.538694 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.538639 2565 scope.go:117] "RemoveContainer" containerID="de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019" Jul 15 22:52:13.547781 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.547762 2565 scope.go:117] "RemoveContainer" containerID="de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019" Jul 15 22:52:13.548032 ip-10-0-130-194 kubenswrapper[2565]: E0715 22:52:13.548015 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019\": container with ID starting with de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019 not found: ID does not exist" containerID="de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019" Jul 15 22:52:13.548095 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.548045 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019"} err="failed to get container status \"de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019\": rpc error: code = NotFound desc = could not find container \"de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019\": container with ID starting with de16a146481f900f4a8df555f4b9d8013add05a202d7d6e5740fa3ad0f429019 not found: ID does not exist" Jul 15 22:52:13.560338 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.560310 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8"] Jul 15 22:52:13.564623 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.564599 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpgh8"] Jul 15 22:52:13.585311 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:13.585278 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c9f460-21c2-4ef6-913f-7c0e2734d0e4" path="/var/lib/kubelet/pods/e6c9f460-21c2-4ef6-913f-7c0e2734d0e4/volumes" Jul 15 22:52:40.264686 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:52:40.264598 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:53:11.784577 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.784539 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz"] Jul 15 22:53:11.785029 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.784880 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c9f460-21c2-4ef6-913f-7c0e2734d0e4" containerName="manager" Jul 15 22:53:11.785029 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.784891 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c9f460-21c2-4ef6-913f-7c0e2734d0e4" containerName="manager" Jul 15 22:53:11.785029 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.784973 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c9f460-21c2-4ef6-913f-7c0e2734d0e4" containerName="manager" Jul 15 22:53:11.789049 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.789030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:11.792002 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.791981 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wt2v9\"" Jul 15 22:53:11.799663 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.799637 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz"] Jul 15 22:53:11.888802 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.888762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bpgpz\" (UID: \"2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:11.888978 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.888826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l649v\" (UniqueName: \"kubernetes.io/projected/2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab-kube-api-access-l649v\") pod \"kuadrant-operator-controller-manager-86878c769f-bpgpz\" (UID: \"2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:11.989321 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.989268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bpgpz\" (UID: \"2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:11.989499 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.989354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l649v\" (UniqueName: \"kubernetes.io/projected/2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab-kube-api-access-l649v\") pod \"kuadrant-operator-controller-manager-86878c769f-bpgpz\" (UID: \"2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:11.989681 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.989664 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bpgpz\" (UID: \"2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:11.998776 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:11.998743 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l649v\" (UniqueName: \"kubernetes.io/projected/2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab-kube-api-access-l649v\") pod \"kuadrant-operator-controller-manager-86878c769f-bpgpz\" (UID: \"2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:12.099874 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:12.099770 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:12.253374 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:12.253345 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz"] Jul 15 22:53:12.256151 ip-10-0-130-194 kubenswrapper[2565]: W0715 22:53:12.256124 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfde4da_cf4f_4ea8_822b_bc7d5cc6beab.slice/crio-6f68324d2a857e158b733c1ab6c59e9a8f45e675a9bd0dd3b90fded85d10eb30 WatchSource:0}: Error finding container 6f68324d2a857e158b733c1ab6c59e9a8f45e675a9bd0dd3b90fded85d10eb30: Status 404 returned error can't find the container with id 6f68324d2a857e158b733c1ab6c59e9a8f45e675a9bd0dd3b90fded85d10eb30 Jul 15 22:53:12.804082 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:12.804038 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" event={"ID":"2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab","Type":"ContainerStarted","Data":"4e7737f389f3cedef489995a4f0944de58bdf554ef9fa9c9f3ed0e452d272d94"} Jul 15 22:53:12.804530 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:12.804103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" event={"ID":"2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab","Type":"ContainerStarted","Data":"6f68324d2a857e158b733c1ab6c59e9a8f45e675a9bd0dd3b90fded85d10eb30"} Jul 15 22:53:12.804530 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:12.804256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:12.824229 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:12.824157 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" podStartSLOduration=1.8241424899999998 podStartE2EDuration="1.82414249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:53:12.822880494 +0000 UTC m=+1203.763083027" watchObservedRunningTime="2026-07-15 22:53:12.82414249 +0000 UTC m=+1203.764345040" Jul 15 22:53:23.811294 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:23.811262 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpgpz" Jul 15 22:53:35.158052 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:35.158014 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:53:44.647019 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:44.646978 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:53:58.355152 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:53:58.355103 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:54:10.952456 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:54:10.952354 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:54:20.562260 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:54:20.562222 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:54:31.745237 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:54:31.745206 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:54:40.950659 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:54:40.950620 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:54:52.144187 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:54:52.144121 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:55:03.147134 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:55:03.147094 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:55:11.747644 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:55:11.747600 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:55:24.046522 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:55:24.046482 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:55:44.149417 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:55:44.149333 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:55:53.251190 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:55:53.251143 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:55:55.573923 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:55:55.573887 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:56:02.957441 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:56:02.957401 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:56:14.553783 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:56:14.553743 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:56:41.447390 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:56:41.447337 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:56:51.154440 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:56:51.154396 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:57:01.363864 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:57:01.363770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:57:10.963578 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:57:10.963537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:57:20.651998 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:57:20.651961 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:57:31.255635 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:57:31.255597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:57:43.149254 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:57:43.149210 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:57:52.753124 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:57:52.753088 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:58:05.547305 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:58:05.547271 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:58:17.441443 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:58:17.441408 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:58:24.755052 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:58:24.755016 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:58:36.648941 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:58:36.648858 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:58:39.298598 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:58:39.298557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:58:39.384277 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:58:39.384243 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:58:48.359646 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:58:48.359607 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:59:03.043870 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:03.043831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:59:03.755958 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:03.755921 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:11.126407 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:11.126363 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:13.434774 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:13.434738 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:14.249335 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:14.249298 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:59:19.385500 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:19.385456 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:24.456280 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:24.456237 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:27.298606 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:27.298557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:27.395916 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:27.395883 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:27.769372 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:27.769341 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:28.154712 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:28.154678 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:59:33.358428 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:33.358392 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:33.459063 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:33.459028 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:33.552382 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:33.552345 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:33.647996 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:33.647954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:33.734727 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:33.734688 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:33.834841 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:33.834786 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:33.959465 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:33.959369 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:34.186650 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:34.186615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:34.288145 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:34.288047 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:37.340540 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:37.340495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:37.846810 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:37.846770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:59:39.806426 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:39.806388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:39.904602 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:39.904566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:41.815851 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:41.815824 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:45.483894 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:45.483848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:45.543192 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:45.543112 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:45.728077 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:45.728043 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:45.836001 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:45.835908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:45.920657 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:45.920614 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:45.980420 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:45.980387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:46.112550 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:46.112459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:46.212806 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:46.212770 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:46.465217 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:46.465185 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:46.708635 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:46.708595 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:46.805744 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:46.805649 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:48.951183 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:48.951144 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:59:49.192544 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:49.192511 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:51.731246 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:51.731204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:51.824100 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:51.824063 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:54.065309 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:54.065280 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:57.532908 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:57.532862 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:57.596730 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:57.596692 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:57.665659 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:57.665624 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 22:59:57.882296 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:57.882267 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:57.982865 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:57.982819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:58.075429 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:58.075338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:58.274147 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:58.274113 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:58.476112 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:58.476084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 22:59:58.576704 ip-10-0-130-194 kubenswrapper[2565]: I0715 22:59:58.576671 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:00.134611 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:00.134525 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd"] Jul 15 23:00:00.137889 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:00.137870 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd" Jul 15 23:00:00.140589 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:00.140561 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rfpqz\"" Jul 15 23:00:00.145467 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:00.145438 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd"] Jul 15 23:00:00.150777 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:00.150755 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd" Jul 15 23:00:00.294354 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:00.294323 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd"] Jul 15 23:00:00.302845 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:00.302822 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 23:00:00.684724 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:00.684688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd" event={"ID":"ac8a1671-124f-4a85-92be-e2e8b026c7b7","Type":"ContainerStarted","Data":"c7590a1256a52b0c47e379a98e0231356f092bf4f64d0ffc9026af17d0da13de"} Jul 15 23:00:01.006757 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:01.006724 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:03.412458 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:03.412423 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:03.586046 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:03.586006 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:05.075445 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:05.075411 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:05.716297 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:05.716261 2565 generic.go:356] "Generic (PLEG): container finished" podID="ac8a1671-124f-4a85-92be-e2e8b026c7b7" containerID="7d5d69e22cc24216d2e61d2a4af9c5a169827ef1067eb116f412dab0aa8fa8ea" exitCode=0 Jul 15 23:00:05.716483 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:05.716296 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd" event={"ID":"ac8a1671-124f-4a85-92be-e2e8b026c7b7","Type":"ContainerDied","Data":"7d5d69e22cc24216d2e61d2a4af9c5a169827ef1067eb116f412dab0aa8fa8ea"} Jul 15 23:00:06.852811 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:06.852788 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd" Jul 15 23:00:07.726114 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:07.726071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd" event={"ID":"ac8a1671-124f-4a85-92be-e2e8b026c7b7","Type":"ContainerDied","Data":"c7590a1256a52b0c47e379a98e0231356f092bf4f64d0ffc9026af17d0da13de"} Jul 15 23:00:07.726114 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:07.726115 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd" Jul 15 23:00:07.726354 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:07.726133 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7590a1256a52b0c47e379a98e0231356f092bf4f64d0ffc9026af17d0da13de" Jul 15 23:00:09.135375 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:09.135337 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:09.196091 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:09.196053 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:09.618181 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:09.618074 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:09.718099 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:09.718061 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:09.833273 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:09.833236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:11.159269 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:11.159238 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:16.895347 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:16.895305 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:17.012600 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:17.012567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:17.110516 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:17.110483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:17.211986 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:17.211904 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:17.585481 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:17.585395 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:20.719510 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:20.719470 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:43.547763 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:43.547727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:49.171302 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:49.171267 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:49.228895 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:49.228861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:49.316360 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:49.316327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:00:49.416234 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:00:49.416190 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:05.239215 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:05.239154 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:11.017939 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:11.017905 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:11.125662 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:11.125632 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:11.218424 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:11.218393 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:21.881342 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:21.881302 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:22.545535 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:22.545501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:28.243775 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:28.243739 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:28.301897 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:28.301849 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:28.377341 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:28.377292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:28.477213 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:28.477181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:28.577667 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:28.577578 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:29.255483 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:29.255446 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:29.356432 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:29.356397 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:29.452657 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:29.452626 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:31.032310 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:31.032218 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:31.953256 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:31.953218 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:01:36.707643 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:36.707603 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:36.808621 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:36.808575 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:37.361808 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:37.361769 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:41.948768 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:41.948728 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:01:43.111880 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:43.111843 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:43.594611 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:43.594572 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:43.676122 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:43.676088 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:43.878737 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:43.878708 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:45.623786 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:45.623750 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:45.724219 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:45.724176 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:46.020383 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:46.020352 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:46.120183 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:46.120141 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:46.222411 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:46.222376 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:50.852217 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:50.852147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:51.454904 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:51.454866 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:51.554750 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:51.554709 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:51.883412 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:51.883385 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:53.352730 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:53.352693 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:01:57.491031 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:57.490995 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:57.590490 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:57.590460 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:01:58.181402 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:01:58.181370 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:04.125515 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:04.125472 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:04.226372 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:04.226340 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:04.440486 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:04.440453 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:04.627121 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:04.627081 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:04.739306 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:04.739221 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:04.840333 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:04.840296 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:04.961265 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:04.961230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:07.250187 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:07.250141 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:02:10.577544 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:10.577510 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:10.678560 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:10.678524 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:10.778811 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:10.778761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:10.924612 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:10.924571 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:16.571978 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:16.571941 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:16.726361 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:16.726323 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:24.086518 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:24.086465 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:24.182470 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:24.182438 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:24.282418 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:24.282374 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:24.957538 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:24.957501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:25.058955 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:25.058914 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:25.155658 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:25.155623 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:26.460342 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:26.460306 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:28.953770 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:28.953734 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:02:32.423443 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:32.423404 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:32.525826 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:32.525789 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:33.097641 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:33.097610 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:38.769836 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:38.769802 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:38.832435 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:38.832393 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:38.905385 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:38.905346 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:39.000524 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:39.000483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:40.654192 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:40.654144 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:02:49.471207 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:49.471137 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:49.795477 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:49.795396 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:56.294855 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:56.294808 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:02:56.397229 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:02:56.397147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:10.741765 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:10.741725 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:16.864408 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:16.864371 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:16.965560 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:16.965499 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:17.249228 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:17.249191 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:17.580241 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:17.580130 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:17.676665 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:17.676629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:19.834411 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:19.834361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:20.085386 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:20.085265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:20.178562 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:20.178529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:20.592835 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:20.592796 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:20.694031 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:20.693994 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:25.298011 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:25.297975 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:25.633129 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:25.633097 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:25.935605 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:25.935514 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:26.310699 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:26.310577 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:27.798088 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:27.798048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:27.919137 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:27.919104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:28.198702 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:28.198660 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:28.349806 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:28.349764 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:28.763262 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:28.763232 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:28.862825 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:28.862788 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:29.756186 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:29.756142 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:03:33.298880 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:33.298825 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:41.150526 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.150491 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc"] Jul 15 23:03:41.150975 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.150957 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8a1671-124f-4a85-92be-e2e8b026c7b7" containerName="cleanup" Jul 15 23:03:41.151034 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.150980 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8a1671-124f-4a85-92be-e2e8b026c7b7" containerName="cleanup" Jul 15 23:03:41.151072 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.151063 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8a1671-124f-4a85-92be-e2e8b026c7b7" containerName="cleanup" Jul 15 23:03:41.154399 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.154365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.159805 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.159783 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-83ad46\"/\"default-dockercfg-h58nz\"" Jul 15 23:03:41.159967 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.159899 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-83ad46\"/\"openshift-service-ca.crt\"" Jul 15 23:03:41.162107 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.161556 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-83ad46\"/\"kube-root-ca.crt\"" Jul 15 23:03:41.162107 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.161556 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-83ad46\"/\"test-model-83ad46-kserve-self-signed-certs\"" Jul 15 23:03:41.171676 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.171650 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc"] Jul 15 23:03:41.196581 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.196545 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:41.255563 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.255525 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-dshm\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.255809 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.255577 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnw9w\" (UniqueName: \"kubernetes.io/projected/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-kube-api-access-cnw9w\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.255809 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.255599 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tls-certs\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.255809 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.255714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-home\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.255809 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.255757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-model-cache\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.255809 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.255774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tmp-dir\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.356957 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.356913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-home\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.356957 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.356963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-model-cache\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.357224 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.356982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tmp-dir\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.357224 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.357018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-dshm\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.357224 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.357052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnw9w\" (UniqueName: \"kubernetes.io/projected/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-kube-api-access-cnw9w\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.357224 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.357076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tls-certs\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.357396 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.357366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-model-cache\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.357454 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.357424 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-home\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.357600 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.357579 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tmp-dir\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.359538 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.359515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-dshm\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.359799 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.359783 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tls-certs\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.369113 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.369083 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnw9w\" (UniqueName: \"kubernetes.io/projected/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-kube-api-access-cnw9w\") pod \"test-model-83ad46-kserve-9c856ff7c-cdmrc\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.469957 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.469863 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:41.610123 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.610053 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc"] Jul 15 23:03:41.712949 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.712901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" event={"ID":"ef9e27af-b74e-4a56-b047-ec1cbfe8978e","Type":"ContainerStarted","Data":"937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7"} Jul 15 23:03:41.713124 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.712958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" event={"ID":"ef9e27af-b74e-4a56-b047-ec1cbfe8978e","Type":"ContainerStarted","Data":"d0f5755d5647bb6e2899c19da1b38cfe458604179aa2fc43920682d09ed22d67"} Jul 15 23:03:41.735908 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.735802 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" podStartSLOduration=0.735786445 podStartE2EDuration="735.786445ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:03:41.735149853 +0000 UTC m=+1832.675352403" watchObservedRunningTime="2026-07-15 23:03:41.735786445 +0000 UTC m=+1832.675988994" Jul 15 23:03:41.865343 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:41.865305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:03:42.481255 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:42.481219 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:42.587750 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:42.587718 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:49.730636 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:49.730605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:49.833469 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:49.833431 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:03:51.470746 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:51.470714 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:51.471222 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:51.471056 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:51.483290 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:51.483264 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:51.770731 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:51.770653 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:03:56.755617 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:03:56.755572 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:04:07.466627 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.466594 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb"] Jul 15 23:04:07.470343 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.470321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.473593 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.473570 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2b485c\"/\"kube-root-ca.crt\"" Jul 15 23:04:07.475150 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.475124 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2b485c\"/\"default-dockercfg-xv8r6\"" Jul 15 23:04:07.475283 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.475233 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2b485c\"/\"test-model-2b485c-kserve-self-signed-certs\"" Jul 15 23:04:07.475283 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.475241 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2b485c\"/\"openshift-service-ca.crt\"" Jul 15 23:04:07.481464 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.481438 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb"] Jul 15 23:04:07.495821 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.495786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-model-cache\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.495985 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.495869 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-tmp-dir\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.495985 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.495891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-home\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.495985 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.495917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e8a726-9593-4f8d-a783-23dd79d8a734-tls-certs\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.495985 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.495951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckmwl\" (UniqueName: \"kubernetes.io/projected/18e8a726-9593-4f8d-a783-23dd79d8a734-kube-api-access-ckmwl\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.496214 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.496000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-dshm\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.513425 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.513389 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:04:07.597355 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.597321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-tmp-dir\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.597568 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.597461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-home\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.597568 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.597493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e8a726-9593-4f8d-a783-23dd79d8a734-tls-certs\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.597568 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.597522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckmwl\" (UniqueName: \"kubernetes.io/projected/18e8a726-9593-4f8d-a783-23dd79d8a734-kube-api-access-ckmwl\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.597743 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.597575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-dshm\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.597743 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.597664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-model-cache\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.597896 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.597869 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-tmp-dir\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.598064 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.598015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-model-cache\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.598518 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.598471 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-home\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.600939 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.600911 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-dshm\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.601300 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.601282 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e8a726-9593-4f8d-a783-23dd79d8a734-tls-certs\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.616285 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.616266 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckmwl\" (UniqueName: \"kubernetes.io/projected/18e8a726-9593-4f8d-a783-23dd79d8a734-kube-api-access-ckmwl\") pod \"test-model-2b485c-kserve-568b549869-gndwb\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.781410 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.781322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:07.938224 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:07.938199 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb"] Jul 15 23:04:07.940713 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:04:07.940691 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18e8a726_9593_4f8d_a783_23dd79d8a734.slice/crio-7ec6b364291dbb92f3b841e0add254ea1a575ef0df56a99985584ebc03da0e74 WatchSource:0}: Error finding container 7ec6b364291dbb92f3b841e0add254ea1a575ef0df56a99985584ebc03da0e74: Status 404 returned error can't find the container with id 7ec6b364291dbb92f3b841e0add254ea1a575ef0df56a99985584ebc03da0e74 Jul 15 23:04:08.844370 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:08.844328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" event={"ID":"18e8a726-9593-4f8d-a783-23dd79d8a734","Type":"ContainerStarted","Data":"508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41"} Jul 15 23:04:08.844370 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:08.844368 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" event={"ID":"18e8a726-9593-4f8d-a783-23dd79d8a734","Type":"ContainerStarted","Data":"7ec6b364291dbb92f3b841e0add254ea1a575ef0df56a99985584ebc03da0e74"} Jul 15 23:04:08.863371 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:08.863311 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" podStartSLOduration=1.863292343 podStartE2EDuration="1.863292343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:04:08.862716759 +0000 UTC m=+1859.802919311" watchObservedRunningTime="2026-07-15 23:04:08.863292343 +0000 UTC m=+1859.803494924" Jul 15 23:04:14.576402 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:14.576350 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:04:17.781880 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:17.781837 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:17.782336 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:17.781907 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:17.794985 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:17.794953 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:17.900222 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:17.900194 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:26.536747 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:26.536706 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc"] Jul 15 23:04:26.561740 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:26.561701 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:04:27.568016 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:27.567976 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb"] Jul 15 23:04:27.590925 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:27.590894 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:04:28.023316 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:28.023279 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:04:33.103453 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:33.103362 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:04:34.550430 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:34.550388 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:04:41.557657 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.557590 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" podUID="ef9e27af-b74e-4a56-b047-ec1cbfe8978e" containerName="main" containerID="cri-o://937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7" gracePeriod=45 Jul 15 23:04:41.808978 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.808918 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:04:41.911920 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.911883 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tmp-dir\") pod \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " Jul 15 23:04:41.912105 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.911934 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tls-certs\") pod \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " Jul 15 23:04:41.912105 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.911968 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnw9w\" (UniqueName: \"kubernetes.io/projected/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-kube-api-access-cnw9w\") pod \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " Jul 15 23:04:41.912105 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.911987 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-model-cache\") pod \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " Jul 15 23:04:41.912105 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.912022 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-home\") pod \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " Jul 15 23:04:41.912105 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.912051 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-dshm\") pod \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\" (UID: \"ef9e27af-b74e-4a56-b047-ec1cbfe8978e\") " Jul 15 23:04:41.912440 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.912227 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef9e27af-b74e-4a56-b047-ec1cbfe8978e" (UID: "ef9e27af-b74e-4a56-b047-ec1cbfe8978e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:04:41.912440 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.912327 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-home" (OuterVolumeSpecName: "home") pod "ef9e27af-b74e-4a56-b047-ec1cbfe8978e" (UID: "ef9e27af-b74e-4a56-b047-ec1cbfe8978e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:04:41.912440 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.912358 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef9e27af-b74e-4a56-b047-ec1cbfe8978e" (UID: "ef9e27af-b74e-4a56-b047-ec1cbfe8978e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:04:41.912440 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.912394 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:41.914466 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.914441 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-dshm" (OuterVolumeSpecName: "dshm") pod "ef9e27af-b74e-4a56-b047-ec1cbfe8978e" (UID: "ef9e27af-b74e-4a56-b047-ec1cbfe8978e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:04:41.914559 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.914536 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-kube-api-access-cnw9w" (OuterVolumeSpecName: "kube-api-access-cnw9w") pod "ef9e27af-b74e-4a56-b047-ec1cbfe8978e" (UID: "ef9e27af-b74e-4a56-b047-ec1cbfe8978e"). InnerVolumeSpecName "kube-api-access-cnw9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:04:41.914597 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:41.914561 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef9e27af-b74e-4a56-b047-ec1cbfe8978e" (UID: "ef9e27af-b74e-4a56-b047-ec1cbfe8978e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:04:42.000427 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.000383 2565 generic.go:356] "Generic (PLEG): container finished" podID="ef9e27af-b74e-4a56-b047-ec1cbfe8978e" containerID="937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7" exitCode=0 Jul 15 23:04:42.000427 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.000429 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" Jul 15 23:04:42.000708 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.000467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" event={"ID":"ef9e27af-b74e-4a56-b047-ec1cbfe8978e","Type":"ContainerDied","Data":"937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7"} Jul 15 23:04:42.000708 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.000517 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" event={"ID":"ef9e27af-b74e-4a56-b047-ec1cbfe8978e","Type":"ContainerDied","Data":"d0f5755d5647bb6e2899c19da1b38cfe458604179aa2fc43920682d09ed22d67"} Jul 15 23:04:42.000708 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.000540 2565 scope.go:117] "RemoveContainer" containerID="937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7" Jul 15 23:04:42.009829 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.009804 2565 scope.go:117] "RemoveContainer" containerID="937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7" Jul 15 23:04:42.010090 ip-10-0-130-194 kubenswrapper[2565]: E0715 23:04:42.010071 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7\": container with ID starting with 937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7 not found: ID does not exist" containerID="937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7" Jul 15 23:04:42.010147 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.010100 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7"} err="failed to get container status \"937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7\": rpc error: code = NotFound desc = could not find container \"937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7\": container with ID starting with 937b916dee370b43d8146f253278fceefca01898d1b73c831276b3f38a83eaa7 not found: ID does not exist" Jul 15 23:04:42.012868 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.012849 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:42.012868 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.012867 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:42.012973 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.012875 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:42.012973 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.012883 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:42.012973 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.012892 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnw9w\" (UniqueName: \"kubernetes.io/projected/ef9e27af-b74e-4a56-b047-ec1cbfe8978e-kube-api-access-cnw9w\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:42.023617 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.023592 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc"] Jul 15 23:04:42.027886 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.027858 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc"] Jul 15 23:04:42.589316 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.589261 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" podUID="18e8a726-9593-4f8d-a783-23dd79d8a734" containerName="main" containerID="cri-o://508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41" gracePeriod=45 Jul 15 23:04:42.842773 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:42.842713 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:43.007780 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.007740 2565 generic.go:356] "Generic (PLEG): container finished" podID="18e8a726-9593-4f8d-a783-23dd79d8a734" containerID="508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41" exitCode=0 Jul 15 23:04:43.007930 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.007799 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" Jul 15 23:04:43.007930 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.007824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" event={"ID":"18e8a726-9593-4f8d-a783-23dd79d8a734","Type":"ContainerDied","Data":"508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41"} Jul 15 23:04:43.007930 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.007869 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb" event={"ID":"18e8a726-9593-4f8d-a783-23dd79d8a734","Type":"ContainerDied","Data":"7ec6b364291dbb92f3b841e0add254ea1a575ef0df56a99985584ebc03da0e74"} Jul 15 23:04:43.007930 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.007889 2565 scope.go:117] "RemoveContainer" containerID="508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41" Jul 15 23:04:43.016851 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.016831 2565 scope.go:117] "RemoveContainer" containerID="508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41" Jul 15 23:04:43.017118 ip-10-0-130-194 kubenswrapper[2565]: E0715 23:04:43.017092 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41\": container with ID starting with 508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41 not found: ID does not exist" containerID="508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41" Jul 15 23:04:43.017213 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.017129 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41"} err="failed to get container status \"508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41\": rpc error: code = NotFound desc = could not find container \"508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41\": container with ID starting with 508115b8a00b082d909b52d0584648b54a1ef714e8104d6275cd56456474dd41 not found: ID does not exist" Jul 15 23:04:43.022426 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.022410 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckmwl\" (UniqueName: \"kubernetes.io/projected/18e8a726-9593-4f8d-a783-23dd79d8a734-kube-api-access-ckmwl\") pod \"18e8a726-9593-4f8d-a783-23dd79d8a734\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " Jul 15 23:04:43.022481 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.022467 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-dshm\") pod \"18e8a726-9593-4f8d-a783-23dd79d8a734\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " Jul 15 23:04:43.022652 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.022637 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e8a726-9593-4f8d-a783-23dd79d8a734-tls-certs\") pod \"18e8a726-9593-4f8d-a783-23dd79d8a734\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " Jul 15 23:04:43.022880 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.022675 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-model-cache\") pod \"18e8a726-9593-4f8d-a783-23dd79d8a734\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " Jul 15 23:04:43.022880 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.022698 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-tmp-dir\") pod \"18e8a726-9593-4f8d-a783-23dd79d8a734\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " Jul 15 23:04:43.022880 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.022784 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-home\") pod \"18e8a726-9593-4f8d-a783-23dd79d8a734\" (UID: \"18e8a726-9593-4f8d-a783-23dd79d8a734\") " Jul 15 23:04:43.023040 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.022927 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-model-cache" (OuterVolumeSpecName: "model-cache") pod "18e8a726-9593-4f8d-a783-23dd79d8a734" (UID: "18e8a726-9593-4f8d-a783-23dd79d8a734"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:04:43.023040 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.023003 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18e8a726-9593-4f8d-a783-23dd79d8a734" (UID: "18e8a726-9593-4f8d-a783-23dd79d8a734"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:04:43.023142 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.023049 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-home" (OuterVolumeSpecName: "home") pod "18e8a726-9593-4f8d-a783-23dd79d8a734" (UID: "18e8a726-9593-4f8d-a783-23dd79d8a734"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:04:43.023142 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.023106 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:43.023246 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.023152 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:43.025091 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.025064 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e8a726-9593-4f8d-a783-23dd79d8a734-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18e8a726-9593-4f8d-a783-23dd79d8a734" (UID: "18e8a726-9593-4f8d-a783-23dd79d8a734"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:04:43.025091 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.025159 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e8a726-9593-4f8d-a783-23dd79d8a734-kube-api-access-ckmwl" (OuterVolumeSpecName: "kube-api-access-ckmwl") pod "18e8a726-9593-4f8d-a783-23dd79d8a734" (UID: "18e8a726-9593-4f8d-a783-23dd79d8a734"). InnerVolumeSpecName "kube-api-access-ckmwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:04:43.025091 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.025196 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-dshm" (OuterVolumeSpecName: "dshm") pod "18e8a726-9593-4f8d-a783-23dd79d8a734" (UID: "18e8a726-9593-4f8d-a783-23dd79d8a734"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:04:43.124524 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.124430 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:43.124524 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.124464 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckmwl\" (UniqueName: \"kubernetes.io/projected/18e8a726-9593-4f8d-a783-23dd79d8a734-kube-api-access-ckmwl\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:43.124524 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.124477 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e8a726-9593-4f8d-a783-23dd79d8a734-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:43.124524 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.124489 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e8a726-9593-4f8d-a783-23dd79d8a734-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:04:43.330818 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.330783 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb"] Jul 15 23:04:43.332463 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.332442 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2b485c/test-model-2b485c-kserve-568b549869-gndwb"] Jul 15 23:04:43.585153 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.585123 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e8a726-9593-4f8d-a783-23dd79d8a734" path="/var/lib/kubelet/pods/18e8a726-9593-4f8d-a783-23dd79d8a734/volumes" Jul 15 23:04:43.585542 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:43.585528 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9e27af-b74e-4a56-b047-ec1cbfe8978e" path="/var/lib/kubelet/pods/ef9e27af-b74e-4a56-b047-ec1cbfe8978e/volumes" Jul 15 23:04:46.761290 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:04:46.761245 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-83ad46/test-model-83ad46-kserve-9c856ff7c-cdmrc" podUID="ef9e27af-b74e-4a56-b047-ec1cbfe8978e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Jul 15 23:05:09.887558 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:09.887526 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:10.165246 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:10.165214 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:10.468375 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:10.468281 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:10.877679 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:10.877635 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:14.153063 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:14.153020 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:05:16.494128 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:16.494090 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:16.604024 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:16.603969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:16.745079 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:16.744992 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:16.968419 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:16.968382 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:17.298881 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:17.298833 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:17.660312 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:17.660281 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:17.834468 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:17.834432 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:19.436952 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:19.436923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:19.538445 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:19.538404 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:19.787242 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:19.787132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:19.885242 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:19.885206 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:20.168486 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:20.168452 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:20.274931 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:20.274890 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:24.319691 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:24.319654 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:24.376044 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:24.376005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:24.951941 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:24.951901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:25.389001 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:25.388966 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:25.682775 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:25.682686 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:25.992314 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:25.992228 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:27.470589 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:27.470558 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:27.559714 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:27.559682 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:27.620954 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:27.620924 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:27.721958 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:27.721864 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:28.127745 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:28.127720 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:28.233374 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:28.233336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:29.065192 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:29.065113 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:05:33.048763 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:33.048711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:40.991490 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:40.991455 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:41.096506 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:41.096463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:41.215097 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:41.215063 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:42.365356 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:42.365322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:43.649279 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:43.649241 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:05:46.918270 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:46.918232 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:48.868447 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:48.868412 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:48.968282 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:48.968246 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:50.868026 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:50.867968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:52.366512 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:52.366478 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:52.431574 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:52.431540 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:52.662837 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:52.662806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:52.892224 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:52.892133 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:53.327979 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:53.327940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:05:58.045338 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:58.045305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:05:58.988086 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:05:58.988053 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:04.535000 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:04.534912 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:04.635346 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:04.635309 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:04.829782 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:04.829704 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:05.049655 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:05.049623 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:05.412543 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:05.412503 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:05.759209 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:05.759110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:05.886708 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:05.886675 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:07.747749 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:07.747710 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:08.061260 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:08.061153 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:06:08.972789 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:08.972757 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:13.243615 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:13.243574 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:13.562279 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:13.562196 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:13.850620 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:13.850527 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:14.300386 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:14.300338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:15.792514 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:15.792477 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:15.876877 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:15.876838 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:15.946396 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:15.946348 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:16.343929 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:16.343893 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:16.446607 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:16.446575 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:21.271390 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:21.271357 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:24.754570 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:24.754533 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:06:29.121457 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:29.121410 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:29.221492 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:29.221434 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:37.038458 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.038415 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6"] Jul 15 23:06:37.038958 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.038929 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e8a726-9593-4f8d-a783-23dd79d8a734" containerName="main" Jul 15 23:06:37.038958 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.038948 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e8a726-9593-4f8d-a783-23dd79d8a734" containerName="main" Jul 15 23:06:37.039076 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.038970 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9e27af-b74e-4a56-b047-ec1cbfe8978e" containerName="main" Jul 15 23:06:37.039076 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.038980 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9e27af-b74e-4a56-b047-ec1cbfe8978e" containerName="main" Jul 15 23:06:37.039076 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.039069 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9e27af-b74e-4a56-b047-ec1cbfe8978e" containerName="main" Jul 15 23:06:37.039262 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.039081 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e8a726-9593-4f8d-a783-23dd79d8a734" containerName="main" Jul 15 23:06:37.042223 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.042202 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.044882 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.044857 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ab95dd\"/\"kube-root-ca.crt\"" Jul 15 23:06:37.044882 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.044873 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ab95dd\"/\"openshift-service-ca.crt\"" Jul 15 23:06:37.046268 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.046244 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ab95dd\"/\"default-dockercfg-47rcn\"" Jul 15 23:06:37.046368 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.046334 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ab95dd\"/\"auth-test-model-ab95dd-kserve-self-signed-certs\"" Jul 15 23:06:37.051380 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.051358 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6"] Jul 15 23:06:37.114362 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.114320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-home\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.114540 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.114384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-dshm\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.114540 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.114464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-tmp-dir\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.114540 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.114489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872b1caf-5673-4a05-99da-acf5b868d2bd-tls-certs\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.114540 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.114523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-model-cache\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.114680 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.114553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/872b1caf-5673-4a05-99da-acf5b868d2bd-kube-api-access-l94dj\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.215913 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.215872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-model-cache\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.215913 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.215912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/872b1caf-5673-4a05-99da-acf5b868d2bd-kube-api-access-l94dj\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.216211 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.215970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-home\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.216211 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.216020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-dshm\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.216211 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.216073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-tmp-dir\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.216211 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.216096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872b1caf-5673-4a05-99da-acf5b868d2bd-tls-certs\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.216419 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.216385 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-model-cache\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.216419 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.216401 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-home\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.216528 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.216503 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-tmp-dir\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.218476 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.218450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-dshm\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.218874 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.218853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872b1caf-5673-4a05-99da-acf5b868d2bd-tls-certs\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.228953 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.228924 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/872b1caf-5673-4a05-99da-acf5b868d2bd-kube-api-access-l94dj\") pod \"auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.244369 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.244330 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:37.352518 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.352424 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:37.493799 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.493776 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6"] Jul 15 23:06:37.496405 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:06:37.496369 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod872b1caf_5673_4a05_99da_acf5b868d2bd.slice/crio-84d56000523c0c3397ba5c51b6a7e50b61fcd46a4baea482b64e6ba6e2d7bdba WatchSource:0}: Error finding container 84d56000523c0c3397ba5c51b6a7e50b61fcd46a4baea482b64e6ba6e2d7bdba: Status 404 returned error can't find the container with id 84d56000523c0c3397ba5c51b6a7e50b61fcd46a4baea482b64e6ba6e2d7bdba Jul 15 23:06:37.538861 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.538825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" event={"ID":"872b1caf-5673-4a05-99da-acf5b868d2bd","Type":"ContainerStarted","Data":"84d56000523c0c3397ba5c51b6a7e50b61fcd46a4baea482b64e6ba6e2d7bdba"} Jul 15 23:06:37.742064 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:37.742022 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:38.179377 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:38.179339 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:06:38.545228 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:38.545118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" event={"ID":"872b1caf-5673-4a05-99da-acf5b868d2bd","Type":"ContainerStarted","Data":"6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e"} Jul 15 23:06:38.562381 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:38.562330 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" podStartSLOduration=1.562309608 podStartE2EDuration="1.562309608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:06:38.561251305 +0000 UTC m=+2009.501453856" watchObservedRunningTime="2026-07-15 23:06:38.562309608 +0000 UTC m=+2009.502512160" Jul 15 23:06:41.151624 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:41.151588 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:06:47.352787 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:47.352741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:47.352787 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:47.352799 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:47.365360 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:47.365331 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:06:47.596914 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:06:47.596881 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:07:13.865461 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:13.865423 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t"] Jul 15 23:07:13.868743 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:13.868727 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:13.872889 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:13.872865 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"auth-test-model-c6dbd3-kserve-self-signed-certs\"" Jul 15 23:07:13.872889 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:13.872881 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"default-dockercfg-h52rw\"" Jul 15 23:07:13.873117 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:13.872872 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"kube-root-ca.crt\"" Jul 15 23:07:13.873117 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:13.872872 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"openshift-service-ca.crt\"" Jul 15 23:07:13.879240 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:13.879219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t"] Jul 15 23:07:13.889692 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:13.889660 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:07:14.035667 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.035623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tls-certs\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.035857 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.035676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-home\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.035857 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.035704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tmp-dir\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.035857 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.035763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd588\" (UniqueName: \"kubernetes.io/projected/31f8ac44-f160-4b0a-b8f2-5b18e560755c-kube-api-access-fd588\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.035857 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.035814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-model-cache\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.036029 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.035869 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-dshm\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137151 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tls-certs\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137385 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-home\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137385 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tmp-dir\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137385 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd588\" (UniqueName: \"kubernetes.io/projected/31f8ac44-f160-4b0a-b8f2-5b18e560755c-kube-api-access-fd588\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137385 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-model-cache\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137385 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-dshm\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137644 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tmp-dir\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137699 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137688 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-home\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.137773 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.137754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-model-cache\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.139804 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.139776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-dshm\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.140101 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.140078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tls-certs\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.144787 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.144768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd588\" (UniqueName: \"kubernetes.io/projected/31f8ac44-f160-4b0a-b8f2-5b18e560755c-kube-api-access-fd588\") pod \"auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.178882 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.178845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:14.321853 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.321828 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t"] Jul 15 23:07:14.324231 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:07:14.324199 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31f8ac44_f160_4b0a_b8f2_5b18e560755c.slice/crio-78a9d5f759ae9dfd13353d8072b71c1afd99a3b931fe2fc2e5cbfddef8eed47b WatchSource:0}: Error finding container 78a9d5f759ae9dfd13353d8072b71c1afd99a3b931fe2fc2e5cbfddef8eed47b: Status 404 returned error can't find the container with id 78a9d5f759ae9dfd13353d8072b71c1afd99a3b931fe2fc2e5cbfddef8eed47b Jul 15 23:07:14.707827 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.707790 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" event={"ID":"31f8ac44-f160-4b0a-b8f2-5b18e560755c","Type":"ContainerStarted","Data":"97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc"} Jul 15 23:07:14.708012 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.707839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" event={"ID":"31f8ac44-f160-4b0a-b8f2-5b18e560755c","Type":"ContainerStarted","Data":"78a9d5f759ae9dfd13353d8072b71c1afd99a3b931fe2fc2e5cbfddef8eed47b"} Jul 15 23:07:14.723642 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:14.723594 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" podStartSLOduration=1.72357859 podStartE2EDuration="1.72357859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:07:14.723364026 +0000 UTC m=+2045.663566579" watchObservedRunningTime="2026-07-15 23:07:14.72357859 +0000 UTC m=+2045.663781140" Jul 15 23:07:18.059705 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:18.059666 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:07:24.179018 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:24.178963 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:24.179018 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:24.179021 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:24.191669 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:24.191638 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:24.768085 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:24.768051 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:07:34.451431 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:34.451338 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:07:48.652345 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:48.652305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:07:57.008924 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:57.008889 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:07:57.110341 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:07:57.110304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:08:05.758693 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:08:05.758654 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:08:11.696115 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:08:11.696066 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:08:21.945941 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:08:21.945905 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:08:41.260325 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:08:41.260289 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:09:01.555098 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:01.555007 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:09:05.210987 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:05.210950 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:09:19.500052 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:19.500011 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:09:36.707699 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:36.707658 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6"] Jul 15 23:09:36.721890 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:36.721856 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:09:37.666897 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:37.666841 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t"] Jul 15 23:09:37.679017 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:37.678981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:09:40.357909 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:40.357872 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:09:50.663634 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.663599 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5"] Jul 15 23:09:50.667498 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.667478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.670053 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.670028 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ab95dd\"/\"sub-test-model-ab95dd-kserve-self-signed-certs\"" Jul 15 23:09:50.674588 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.674564 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5"] Jul 15 23:09:50.690323 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.690289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:09:50.816561 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.816519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tmp-dir\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.816561 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.816566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-model-cache\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.816824 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.816708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tls-certs\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.816824 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.816741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-home\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.816824 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.816796 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-dshm\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.816824 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.816816 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzz2p\" (UniqueName: \"kubernetes.io/projected/02de9dd2-f8f2-405a-915d-b6ebf02d124e-kube-api-access-tzz2p\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.917411 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tls-certs\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.917411 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-home\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.917411 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-dshm\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.917701 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzz2p\" (UniqueName: \"kubernetes.io/projected/02de9dd2-f8f2-405a-915d-b6ebf02d124e-kube-api-access-tzz2p\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.917701 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tmp-dir\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.917701 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-model-cache\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.917881 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-home\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.917953 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917903 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-model-cache\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.918011 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.917948 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tmp-dir\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.920275 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.920250 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-dshm\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.920388 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.920320 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tls-certs\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.925891 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.925870 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzz2p\" (UniqueName: \"kubernetes.io/projected/02de9dd2-f8f2-405a-915d-b6ebf02d124e-kube-api-access-tzz2p\") pod \"sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:50.979009 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:50.978969 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:09:51.119648 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:51.119617 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5"] Jul 15 23:09:51.121621 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:09:51.121585 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02de9dd2_f8f2_405a_915d_b6ebf02d124e.slice/crio-afa89fc2681733c276ab8bffdc515646542775cf049223fbabb669c64df4bf72 WatchSource:0}: Error finding container afa89fc2681733c276ab8bffdc515646542775cf049223fbabb669c64df4bf72: Status 404 returned error can't find the container with id afa89fc2681733c276ab8bffdc515646542775cf049223fbabb669c64df4bf72 Jul 15 23:09:51.440677 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:51.440584 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" event={"ID":"02de9dd2-f8f2-405a-915d-b6ebf02d124e","Type":"ContainerStarted","Data":"d34ed807d7f78d41038bff15a97ab4ead56df886b645a7623caca981f4590119"} Jul 15 23:09:51.440677 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:51.440625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" event={"ID":"02de9dd2-f8f2-405a-915d-b6ebf02d124e","Type":"ContainerStarted","Data":"afa89fc2681733c276ab8bffdc515646542775cf049223fbabb669c64df4bf72"} Jul 15 23:09:51.457889 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:51.457840 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" podStartSLOduration=1.457824179 podStartE2EDuration="1.457824179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:09:51.4554217 +0000 UTC m=+2202.395624250" watchObservedRunningTime="2026-07-15 23:09:51.457824179 +0000 UTC m=+2202.398026728" Jul 15 23:09:51.729140 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:51.729042 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" podUID="872b1caf-5673-4a05-99da-acf5b868d2bd" containerName="main" containerID="cri-o://6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e" gracePeriod=45 Jul 15 23:09:51.986953 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:51.986887 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:09:52.129836 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.129798 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-model-cache\") pod \"872b1caf-5673-4a05-99da-acf5b868d2bd\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " Jul 15 23:09:52.130046 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.129871 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872b1caf-5673-4a05-99da-acf5b868d2bd-tls-certs\") pod \"872b1caf-5673-4a05-99da-acf5b868d2bd\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " Jul 15 23:09:52.130046 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.129934 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-dshm\") pod \"872b1caf-5673-4a05-99da-acf5b868d2bd\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " Jul 15 23:09:52.130046 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.129971 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-home\") pod \"872b1caf-5673-4a05-99da-acf5b868d2bd\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " Jul 15 23:09:52.130046 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.129993 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-tmp-dir\") pod \"872b1caf-5673-4a05-99da-acf5b868d2bd\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " Jul 15 23:09:52.130046 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.130016 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/872b1caf-5673-4a05-99da-acf5b868d2bd-kube-api-access-l94dj\") pod \"872b1caf-5673-4a05-99da-acf5b868d2bd\" (UID: \"872b1caf-5673-4a05-99da-acf5b868d2bd\") " Jul 15 23:09:52.130316 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.130136 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "872b1caf-5673-4a05-99da-acf5b868d2bd" (UID: "872b1caf-5673-4a05-99da-acf5b868d2bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:09:52.130316 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.130217 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-home" (OuterVolumeSpecName: "home") pod "872b1caf-5673-4a05-99da-acf5b868d2bd" (UID: "872b1caf-5673-4a05-99da-acf5b868d2bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:09:52.130316 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.130298 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "872b1caf-5673-4a05-99da-acf5b868d2bd" (UID: "872b1caf-5673-4a05-99da-acf5b868d2bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:09:52.130429 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.130377 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:52.130429 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.130397 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:52.130429 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.130421 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:52.132494 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.132458 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-dshm" (OuterVolumeSpecName: "dshm") pod "872b1caf-5673-4a05-99da-acf5b868d2bd" (UID: "872b1caf-5673-4a05-99da-acf5b868d2bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:09:52.132494 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.132487 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872b1caf-5673-4a05-99da-acf5b868d2bd-kube-api-access-l94dj" (OuterVolumeSpecName: "kube-api-access-l94dj") pod "872b1caf-5673-4a05-99da-acf5b868d2bd" (UID: "872b1caf-5673-4a05-99da-acf5b868d2bd"). InnerVolumeSpecName "kube-api-access-l94dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:09:52.132653 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.132494 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872b1caf-5673-4a05-99da-acf5b868d2bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "872b1caf-5673-4a05-99da-acf5b868d2bd" (UID: "872b1caf-5673-4a05-99da-acf5b868d2bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:09:52.231870 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.231828 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872b1caf-5673-4a05-99da-acf5b868d2bd-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:52.231870 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.231862 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872b1caf-5673-4a05-99da-acf5b868d2bd-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:52.231870 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.231877 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/872b1caf-5673-4a05-99da-acf5b868d2bd-kube-api-access-l94dj\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:52.447128 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.447095 2565 generic.go:356] "Generic (PLEG): container finished" podID="872b1caf-5673-4a05-99da-acf5b868d2bd" containerID="6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e" exitCode=0 Jul 15 23:09:52.447329 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.447137 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" Jul 15 23:09:52.447329 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.447197 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" event={"ID":"872b1caf-5673-4a05-99da-acf5b868d2bd","Type":"ContainerDied","Data":"6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e"} Jul 15 23:09:52.447329 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.447240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6" event={"ID":"872b1caf-5673-4a05-99da-acf5b868d2bd","Type":"ContainerDied","Data":"84d56000523c0c3397ba5c51b6a7e50b61fcd46a4baea482b64e6ba6e2d7bdba"} Jul 15 23:09:52.447329 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.447262 2565 scope.go:117] "RemoveContainer" containerID="6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e" Jul 15 23:09:52.456347 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.456332 2565 scope.go:117] "RemoveContainer" containerID="6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e" Jul 15 23:09:52.456609 ip-10-0-130-194 kubenswrapper[2565]: E0715 23:09:52.456587 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e\": container with ID starting with 6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e not found: ID does not exist" containerID="6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e" Jul 15 23:09:52.456678 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.456618 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e"} err="failed to get container status \"6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e\": rpc error: code = NotFound desc = could not find container \"6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e\": container with ID starting with 6e019ed7d18de9335c62658a20b12fd1ffd0512c8c4fd1e2efc96457007d814e not found: ID does not exist" Jul 15 23:09:52.467130 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.467107 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6"] Jul 15 23:09:52.470351 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.470331 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/auth-test-model-ab95dd-kserve-6978f4d6f9-xqvt6"] Jul 15 23:09:52.687947 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.687898 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" podUID="31f8ac44-f160-4b0a-b8f2-5b18e560755c" containerName="main" containerID="cri-o://97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc" gracePeriod=45 Jul 15 23:09:52.953030 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:52.953003 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:09:53.038516 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038486 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd588\" (UniqueName: \"kubernetes.io/projected/31f8ac44-f160-4b0a-b8f2-5b18e560755c-kube-api-access-fd588\") pod \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " Jul 15 23:09:53.038711 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038536 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-home\") pod \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " Jul 15 23:09:53.038759 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038735 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-model-cache\") pod \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " Jul 15 23:09:53.038759 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038751 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-home" (OuterVolumeSpecName: "home") pod "31f8ac44-f160-4b0a-b8f2-5b18e560755c" (UID: "31f8ac44-f160-4b0a-b8f2-5b18e560755c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:09:53.038839 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038808 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tmp-dir\") pod \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " Jul 15 23:09:53.038890 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038839 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tls-certs\") pod \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " Jul 15 23:09:53.038890 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038874 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-dshm\") pod \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\" (UID: \"31f8ac44-f160-4b0a-b8f2-5b18e560755c\") " Jul 15 23:09:53.038975 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038954 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-model-cache" (OuterVolumeSpecName: "model-cache") pod "31f8ac44-f160-4b0a-b8f2-5b18e560755c" (UID: "31f8ac44-f160-4b0a-b8f2-5b18e560755c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:09:53.039012 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.038973 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31f8ac44-f160-4b0a-b8f2-5b18e560755c" (UID: "31f8ac44-f160-4b0a-b8f2-5b18e560755c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:09:53.039208 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.039193 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:53.039284 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.039210 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:53.039284 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.039219 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:53.040964 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.040933 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f8ac44-f160-4b0a-b8f2-5b18e560755c-kube-api-access-fd588" (OuterVolumeSpecName: "kube-api-access-fd588") pod "31f8ac44-f160-4b0a-b8f2-5b18e560755c" (UID: "31f8ac44-f160-4b0a-b8f2-5b18e560755c"). InnerVolumeSpecName "kube-api-access-fd588". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:09:53.041076 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.041046 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31f8ac44-f160-4b0a-b8f2-5b18e560755c" (UID: "31f8ac44-f160-4b0a-b8f2-5b18e560755c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:09:53.041076 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.041063 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-dshm" (OuterVolumeSpecName: "dshm") pod "31f8ac44-f160-4b0a-b8f2-5b18e560755c" (UID: "31f8ac44-f160-4b0a-b8f2-5b18e560755c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:09:53.139814 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.139773 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f8ac44-f160-4b0a-b8f2-5b18e560755c-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:53.139814 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.139807 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f8ac44-f160-4b0a-b8f2-5b18e560755c-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:53.140033 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.139820 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd588\" (UniqueName: \"kubernetes.io/projected/31f8ac44-f160-4b0a-b8f2-5b18e560755c-kube-api-access-fd588\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:09:53.452893 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.452796 2565 generic.go:356] "Generic (PLEG): container finished" podID="31f8ac44-f160-4b0a-b8f2-5b18e560755c" containerID="97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc" exitCode=0 Jul 15 23:09:53.452893 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.452842 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" Jul 15 23:09:53.452893 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.452883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" event={"ID":"31f8ac44-f160-4b0a-b8f2-5b18e560755c","Type":"ContainerDied","Data":"97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc"} Jul 15 23:09:53.453149 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.452925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t" event={"ID":"31f8ac44-f160-4b0a-b8f2-5b18e560755c","Type":"ContainerDied","Data":"78a9d5f759ae9dfd13353d8072b71c1afd99a3b931fe2fc2e5cbfddef8eed47b"} Jul 15 23:09:53.453149 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.452946 2565 scope.go:117] "RemoveContainer" containerID="97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc" Jul 15 23:09:53.462412 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.462394 2565 scope.go:117] "RemoveContainer" containerID="97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc" Jul 15 23:09:53.462703 ip-10-0-130-194 kubenswrapper[2565]: E0715 23:09:53.462683 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc\": container with ID starting with 97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc not found: ID does not exist" containerID="97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc" Jul 15 23:09:53.462770 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.462715 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc"} err="failed to get container status \"97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc\": rpc error: code = NotFound desc = could not find container \"97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc\": container with ID starting with 97fe772df0945bf09f0bd9ceb9395b72379a3e24b334fc6b3273231354ba7adc not found: ID does not exist" Jul 15 23:09:53.473633 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.473598 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t"] Jul 15 23:09:53.475682 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.475661 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/auth-test-model-c6dbd3-kserve-65dd878f9f-2jw4t"] Jul 15 23:09:53.585546 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.585511 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f8ac44-f160-4b0a-b8f2-5b18e560755c" path="/var/lib/kubelet/pods/31f8ac44-f160-4b0a-b8f2-5b18e560755c/volumes" Jul 15 23:09:53.585940 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:53.585927 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872b1caf-5673-4a05-99da-acf5b868d2bd" path="/var/lib/kubelet/pods/872b1caf-5673-4a05-99da-acf5b868d2bd/volumes" Jul 15 23:09:59.448636 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:09:59.448588 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:10:00.979503 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:00.979464 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:10:00.979932 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:00.979540 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:10:00.992355 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:00.992320 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:10:01.510860 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:01.510818 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:10:10.289868 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.289835 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:10:10.752946 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.752912 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7"] Jul 15 23:10:10.753364 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.753346 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f8ac44-f160-4b0a-b8f2-5b18e560755c" containerName="main" Jul 15 23:10:10.753364 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.753365 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f8ac44-f160-4b0a-b8f2-5b18e560755c" containerName="main" Jul 15 23:10:10.753525 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.753396 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872b1caf-5673-4a05-99da-acf5b868d2bd" containerName="main" Jul 15 23:10:10.753525 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.753404 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="872b1caf-5673-4a05-99da-acf5b868d2bd" containerName="main" Jul 15 23:10:10.753525 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.753498 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="872b1caf-5673-4a05-99da-acf5b868d2bd" containerName="main" Jul 15 23:10:10.753525 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.753508 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f8ac44-f160-4b0a-b8f2-5b18e560755c" containerName="main" Jul 15 23:10:10.758373 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.758350 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:10.762596 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.762573 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"openshift-service-ca.crt\"" Jul 15 23:10:10.762775 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.762758 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"default-dockercfg-h52rw\"" Jul 15 23:10:10.762880 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.762799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"sub-test-model-c6dbd3-kserve-self-signed-certs\"" Jul 15 23:10:10.762942 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.762874 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"kube-root-ca.crt\"" Jul 15 23:10:10.767282 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.767260 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7"] Jul 15 23:10:10.850866 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.850826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:10:10.899179 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.899113 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-home\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:10.899179 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.899159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tmp-dir\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:10.899473 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.899234 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tls-certs\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:10.899473 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.899319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-dshm\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:10.899473 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.899345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxz9c\" (UniqueName: \"kubernetes.io/projected/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-kube-api-access-pxz9c\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:10.899473 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:10.899387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-model-cache\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.000480 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.000446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-model-cache\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.000685 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.000561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-home\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.000685 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.000592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tmp-dir\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.000685 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.000642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tls-certs\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.000850 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.000726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-dshm\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.000850 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.000754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxz9c\" (UniqueName: \"kubernetes.io/projected/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-kube-api-access-pxz9c\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.000953 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.000903 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-model-cache\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.001108 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.001090 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tmp-dir\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.001254 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.001238 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-home\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.003274 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.003210 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-dshm\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.003541 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.003523 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tls-certs\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.008777 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.008752 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxz9c\" (UniqueName: \"kubernetes.io/projected/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-kube-api-access-pxz9c\") pod \"sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.068266 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.068232 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:11.206505 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.206467 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7"] Jul 15 23:10:11.210559 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:10:11.210532 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdcfa1e8_6d0d_4cd8_a736_7f5d01af8d61.slice/crio-b46a9da2e359cf89c73cf120dbf665015209750a99754fb5f9a1aa27b3031d8c WatchSource:0}: Error finding container b46a9da2e359cf89c73cf120dbf665015209750a99754fb5f9a1aa27b3031d8c: Status 404 returned error can't find the container with id b46a9da2e359cf89c73cf120dbf665015209750a99754fb5f9a1aa27b3031d8c Jul 15 23:10:11.544849 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.544748 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" event={"ID":"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61","Type":"ContainerStarted","Data":"af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff"} Jul 15 23:10:11.544849 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.544794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" event={"ID":"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61","Type":"ContainerStarted","Data":"b46a9da2e359cf89c73cf120dbf665015209750a99754fb5f9a1aa27b3031d8c"} Jul 15 23:10:11.562797 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:11.562745 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" podStartSLOduration=1.562727446 podStartE2EDuration="1.562727446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:10:11.561115665 +0000 UTC m=+2222.501318215" watchObservedRunningTime="2026-07-15 23:10:11.562727446 +0000 UTC m=+2222.502929996" Jul 15 23:10:20.150397 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:20.150359 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:10:21.068775 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:21.068736 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:21.068775 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:21.068778 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:21.081546 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:21.081501 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:21.608609 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:21.608577 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:10:32.668612 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:32.668479 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:10:32.723273 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:32.723235 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:10:44.764923 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:44.764889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:10:50.591045 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:50.591013 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5"] Jul 15 23:10:50.614477 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:50.614447 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:10:51.440377 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:51.440346 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7"] Jul 15 23:10:51.462208 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:10:51.462101 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:04.058812 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.058776 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx"] Jul 15 23:11:04.062370 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.062351 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.064985 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.064962 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ab95dd\"/\"rate-test-model-ab95dd-kserve-self-signed-certs\"" Jul 15 23:11:04.072692 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.072663 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx"] Jul 15 23:11:04.077612 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.077573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmwk\" (UniqueName: \"kubernetes.io/projected/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-kube-api-access-7hmwk\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.077765 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.077661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-dshm\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.077765 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.077687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-model-cache\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.077765 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.077716 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-home\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.077916 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.077797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tmp-dir\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.077916 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.077829 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tls-certs\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.079610 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.079581 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:04.179026 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.178984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-dshm\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.179026 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.179034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-model-cache\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.179360 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.179071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-home\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.179360 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.179098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tmp-dir\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.179360 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.179120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tls-certs\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.179360 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.179227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmwk\" (UniqueName: \"kubernetes.io/projected/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-kube-api-access-7hmwk\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.179574 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.179556 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-model-cache\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.179624 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.179591 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-home\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.179662 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.179631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tmp-dir\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.181463 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.181432 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-dshm\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.181822 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.181803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tls-certs\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.187532 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.187513 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmwk\" (UniqueName: \"kubernetes.io/projected/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-kube-api-access-7hmwk\") pod \"rate-test-model-ab95dd-kserve-6f49c58875-c7qxx\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.373042 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.372948 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:04.514116 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.514069 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx"] Jul 15 23:11:04.517764 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:11:04.517735 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89edbc6f_e2a5_4f02_a51f_cbcca05914b5.slice/crio-0f84a75cec2339b864bf205a1ae48df24f63b7d7724f42cfc2d933f1c636b728 WatchSource:0}: Error finding container 0f84a75cec2339b864bf205a1ae48df24f63b7d7724f42cfc2d933f1c636b728: Status 404 returned error can't find the container with id 0f84a75cec2339b864bf205a1ae48df24f63b7d7724f42cfc2d933f1c636b728 Jul 15 23:11:04.801319 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.801285 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" event={"ID":"89edbc6f-e2a5-4f02-a51f-cbcca05914b5","Type":"ContainerStarted","Data":"314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9"} Jul 15 23:11:04.801507 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.801327 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" event={"ID":"89edbc6f-e2a5-4f02-a51f-cbcca05914b5","Type":"ContainerStarted","Data":"0f84a75cec2339b864bf205a1ae48df24f63b7d7724f42cfc2d933f1c636b728"} Jul 15 23:11:04.818804 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:04.818749 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" podStartSLOduration=0.818733166 podStartE2EDuration="818.733166ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:11:04.817161625 +0000 UTC m=+2275.757364176" watchObservedRunningTime="2026-07-15 23:11:04.818733166 +0000 UTC m=+2275.758935715" Jul 15 23:11:05.612006 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.611965 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" podUID="02de9dd2-f8f2-405a-915d-b6ebf02d124e" containerName="main" containerID="cri-o://d34ed807d7f78d41038bff15a97ab4ead56df886b645a7623caca981f4590119" gracePeriod=45 Jul 15 23:11:05.810076 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.810035 2565 generic.go:356] "Generic (PLEG): container finished" podID="02de9dd2-f8f2-405a-915d-b6ebf02d124e" containerID="d34ed807d7f78d41038bff15a97ab4ead56df886b645a7623caca981f4590119" exitCode=0 Jul 15 23:11:05.810271 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.810103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" event={"ID":"02de9dd2-f8f2-405a-915d-b6ebf02d124e","Type":"ContainerDied","Data":"d34ed807d7f78d41038bff15a97ab4ead56df886b645a7623caca981f4590119"} Jul 15 23:11:05.873557 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.873491 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:11:05.896035 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.895997 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tls-certs\") pod \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " Jul 15 23:11:05.896263 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.896073 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tmp-dir\") pod \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " Jul 15 23:11:05.896263 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.896097 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-dshm\") pod \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " Jul 15 23:11:05.896263 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.896192 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzz2p\" (UniqueName: \"kubernetes.io/projected/02de9dd2-f8f2-405a-915d-b6ebf02d124e-kube-api-access-tzz2p\") pod \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " Jul 15 23:11:05.896263 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.896231 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-model-cache\") pod \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " Jul 15 23:11:05.896263 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.896253 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-home\") pod \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\" (UID: \"02de9dd2-f8f2-405a-915d-b6ebf02d124e\") " Jul 15 23:11:05.896695 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.896667 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-home" (OuterVolumeSpecName: "home") pod "02de9dd2-f8f2-405a-915d-b6ebf02d124e" (UID: "02de9dd2-f8f2-405a-915d-b6ebf02d124e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:11:05.897141 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.897111 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02de9dd2-f8f2-405a-915d-b6ebf02d124e" (UID: "02de9dd2-f8f2-405a-915d-b6ebf02d124e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:11:05.897333 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.897311 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-model-cache" (OuterVolumeSpecName: "model-cache") pod "02de9dd2-f8f2-405a-915d-b6ebf02d124e" (UID: "02de9dd2-f8f2-405a-915d-b6ebf02d124e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:11:05.899045 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.899022 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-dshm" (OuterVolumeSpecName: "dshm") pod "02de9dd2-f8f2-405a-915d-b6ebf02d124e" (UID: "02de9dd2-f8f2-405a-915d-b6ebf02d124e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:11:05.899045 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.899028 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02de9dd2-f8f2-405a-915d-b6ebf02d124e" (UID: "02de9dd2-f8f2-405a-915d-b6ebf02d124e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:11:05.899635 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.899615 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02de9dd2-f8f2-405a-915d-b6ebf02d124e-kube-api-access-tzz2p" (OuterVolumeSpecName: "kube-api-access-tzz2p") pod "02de9dd2-f8f2-405a-915d-b6ebf02d124e" (UID: "02de9dd2-f8f2-405a-915d-b6ebf02d124e"). InnerVolumeSpecName "kube-api-access-tzz2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:11:05.997882 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.997846 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:05.997882 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.997878 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:05.997882 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.997889 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzz2p\" (UniqueName: \"kubernetes.io/projected/02de9dd2-f8f2-405a-915d-b6ebf02d124e-kube-api-access-tzz2p\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:05.998160 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.997898 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:05.998160 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.997907 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02de9dd2-f8f2-405a-915d-b6ebf02d124e-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:05.998160 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:05.997917 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02de9dd2-f8f2-405a-915d-b6ebf02d124e-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:06.461046 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.461000 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" podUID="bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" containerName="main" containerID="cri-o://af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff" gracePeriod=45 Jul 15 23:11:06.721003 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.720940 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:11:06.805266 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805230 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-home\") pod \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " Jul 15 23:11:06.805483 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805297 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-dshm\") pod \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " Jul 15 23:11:06.805483 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-model-cache\") pod \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " Jul 15 23:11:06.805483 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805364 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tls-certs\") pod \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " Jul 15 23:11:06.805483 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805391 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tmp-dir\") pod \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " Jul 15 23:11:06.805483 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805452 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxz9c\" (UniqueName: \"kubernetes.io/projected/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-kube-api-access-pxz9c\") pod \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\" (UID: \"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61\") " Jul 15 23:11:06.805763 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805521 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-home" (OuterVolumeSpecName: "home") pod "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" (UID: "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:11:06.805763 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805670 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" (UID: "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:11:06.805878 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805799 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:06.805878 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805817 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:06.805962 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.805934 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" (UID: "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:11:06.807698 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.807670 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-kube-api-access-pxz9c" (OuterVolumeSpecName: "kube-api-access-pxz9c") pod "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" (UID: "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61"). InnerVolumeSpecName "kube-api-access-pxz9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:11:06.808116 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.808095 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-dshm" (OuterVolumeSpecName: "dshm") pod "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" (UID: "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:11:06.808218 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.808097 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" (UID: "bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:11:06.817708 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.817678 2565 generic.go:356] "Generic (PLEG): container finished" podID="bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" containerID="af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff" exitCode=0 Jul 15 23:11:06.817810 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.817727 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" Jul 15 23:11:06.817810 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.817769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" event={"ID":"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61","Type":"ContainerDied","Data":"af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff"} Jul 15 23:11:06.817903 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.817810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7" event={"ID":"bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61","Type":"ContainerDied","Data":"b46a9da2e359cf89c73cf120dbf665015209750a99754fb5f9a1aa27b3031d8c"} Jul 15 23:11:06.817903 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.817831 2565 scope.go:117] "RemoveContainer" containerID="af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff" Jul 15 23:11:06.819188 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.819149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" event={"ID":"02de9dd2-f8f2-405a-915d-b6ebf02d124e","Type":"ContainerDied","Data":"afa89fc2681733c276ab8bffdc515646542775cf049223fbabb669c64df4bf72"} Jul 15 23:11:06.819291 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.819191 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5" Jul 15 23:11:06.829624 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.827792 2565 scope.go:117] "RemoveContainer" containerID="af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff" Jul 15 23:11:06.829768 ip-10-0-130-194 kubenswrapper[2565]: E0715 23:11:06.829619 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff\": container with ID starting with af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff not found: ID does not exist" containerID="af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff" Jul 15 23:11:06.829768 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.829661 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff"} err="failed to get container status \"af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff\": rpc error: code = NotFound desc = could not find container \"af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff\": container with ID starting with af10a1fed954ce517007fd82b5b30af11990a6948954ac21b921f8393d0101ff not found: ID does not exist" Jul 15 23:11:06.829768 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.829685 2565 scope.go:117] "RemoveContainer" containerID="d34ed807d7f78d41038bff15a97ab4ead56df886b645a7623caca981f4590119" Jul 15 23:11:06.846050 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.846028 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5"] Jul 15 23:11:06.851907 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.851886 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/sub-test-model-ab95dd-kserve-cdcf7bc7c-spmr5"] Jul 15 23:11:06.864079 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.864056 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7"] Jul 15 23:11:06.867881 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.867858 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/sub-test-model-c6dbd3-kserve-65dfd89d7f-9vtg7"] Jul 15 23:11:06.906699 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.906667 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:06.906699 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.906697 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:06.906699 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.906707 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:06.906923 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:06.906715 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxz9c\" (UniqueName: \"kubernetes.io/projected/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61-kube-api-access-pxz9c\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:11:07.585925 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:07.585892 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02de9dd2-f8f2-405a-915d-b6ebf02d124e" path="/var/lib/kubelet/pods/02de9dd2-f8f2-405a-915d-b6ebf02d124e/volumes" Jul 15 23:11:07.586314 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:07.586300 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" path="/var/lib/kubelet/pods/bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61/volumes" Jul 15 23:11:08.353061 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:08.353022 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:11:14.374114 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:14.374070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:14.374114 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:14.374124 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:14.386695 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:14.386668 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:14.869053 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:14.869022 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:11:30.003950 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:30.003913 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:32.330874 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:32.330831 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:32.847432 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:32.847389 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:32.948662 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:32.948625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:38.157655 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:38.157615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:38.264064 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:38.264024 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:55.862216 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.862175 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75"] Jul 15 23:11:55.862674 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.862651 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" containerName="main" Jul 15 23:11:55.862674 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.862670 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" containerName="main" Jul 15 23:11:55.862804 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.862697 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02de9dd2-f8f2-405a-915d-b6ebf02d124e" containerName="main" Jul 15 23:11:55.862804 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.862706 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02de9dd2-f8f2-405a-915d-b6ebf02d124e" containerName="main" Jul 15 23:11:55.862804 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.862793 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdcfa1e8-6d0d-4cd8-a736-7f5d01af8d61" containerName="main" Jul 15 23:11:55.862804 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.862803 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02de9dd2-f8f2-405a-915d-b6ebf02d124e" containerName="main" Jul 15 23:11:55.866293 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.866271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:55.870141 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.870119 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"kube-root-ca.crt\"" Jul 15 23:11:55.870299 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.870231 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"openshift-service-ca.crt\"" Jul 15 23:11:55.870376 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.870362 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"default-dockercfg-h52rw\"" Jul 15 23:11:55.870493 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.870474 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6dbd3\"/\"rate-test-model-c6dbd3-kserve-self-signed-certs\"" Jul 15 23:11:55.877986 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.877960 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75"] Jul 15 23:11:55.892284 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.892249 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:11:55.935359 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.935324 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-home\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:55.935616 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.935597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d45311c-c505-4936-b997-c4aa5edbd8d6-tls-certs\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:55.935719 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.935707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-model-cache\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:55.935819 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.935807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-dshm\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:55.935956 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.935944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-tmp-dir\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:55.936063 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:55.936049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fftq\" (UniqueName: \"kubernetes.io/projected/6d45311c-c505-4936-b997-c4aa5edbd8d6-kube-api-access-4fftq\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.036875 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.036831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fftq\" (UniqueName: \"kubernetes.io/projected/6d45311c-c505-4936-b997-c4aa5edbd8d6-kube-api-access-4fftq\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.037062 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.036892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-home\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.037062 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.036916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d45311c-c505-4936-b997-c4aa5edbd8d6-tls-certs\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.037062 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.036937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-model-cache\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.037062 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.036955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-dshm\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.037062 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.037002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-tmp-dir\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.037454 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.037428 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-model-cache\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.037522 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.037486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-tmp-dir\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.037574 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.037542 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-home\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.039633 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.039608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-dshm\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.039957 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.039941 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d45311c-c505-4936-b997-c4aa5edbd8d6-tls-certs\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.044893 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.044869 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fftq\" (UniqueName: \"kubernetes.io/projected/6d45311c-c505-4936-b997-c4aa5edbd8d6-kube-api-access-4fftq\") pod \"rate-test-model-c6dbd3-kserve-69d4447d67-krs75\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.177792 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.177728 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:11:56.320259 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:56.320233 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75"] Jul 15 23:11:56.322251 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:11:56.322218 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d45311c_c505_4936_b997_c4aa5edbd8d6.slice/crio-dfd1698771bc7c890c59e5370f06d16af4355a57c200b367c1485867cf230363 WatchSource:0}: Error finding container dfd1698771bc7c890c59e5370f06d16af4355a57c200b367c1485867cf230363: Status 404 returned error can't find the container with id dfd1698771bc7c890c59e5370f06d16af4355a57c200b367c1485867cf230363 Jul 15 23:11:57.050278 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:57.050239 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" event={"ID":"6d45311c-c505-4936-b997-c4aa5edbd8d6","Type":"ContainerStarted","Data":"8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7"} Jul 15 23:11:57.050278 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:57.050283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" event={"ID":"6d45311c-c505-4936-b997-c4aa5edbd8d6","Type":"ContainerStarted","Data":"dfd1698771bc7c890c59e5370f06d16af4355a57c200b367c1485867cf230363"} Jul 15 23:11:57.065349 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:11:57.065284 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" podStartSLOduration=2.065264579 podStartE2EDuration="2.065264579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:11:57.06507628 +0000 UTC m=+2328.005278831" watchObservedRunningTime="2026-07-15 23:11:57.065264579 +0000 UTC m=+2328.005467141" Jul 15 23:12:00.558152 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:00.558065 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:12:06.178514 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:06.178476 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:12:06.179097 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:06.178553 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:12:06.191676 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:06.191649 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:12:07.115086 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:07.115057 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:12:16.563597 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:16.563563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:12:17.760343 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:17.760309 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:12:27.521417 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:27.521380 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:12:35.455222 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:35.455181 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:12:42.373723 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:42.373681 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx"] Jul 15 23:12:42.384691 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:42.384659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:12:43.362226 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:43.362193 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75"] Jul 15 23:12:43.382775 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:43.382740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:12:52.444412 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:52.444370 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:12:57.396513 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.396447 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" podUID="89edbc6f-e2a5-4f02-a51f-cbcca05914b5" containerName="main" containerID="cri-o://314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9" gracePeriod=45 Jul 15 23:12:57.655504 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.655437 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:12:57.688366 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688332 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-model-cache\") pod \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " Jul 15 23:12:57.688560 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688385 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tls-certs\") pod \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " Jul 15 23:12:57.688560 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688408 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tmp-dir\") pod \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " Jul 15 23:12:57.688560 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688437 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hmwk\" (UniqueName: \"kubernetes.io/projected/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-kube-api-access-7hmwk\") pod \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " Jul 15 23:12:57.688560 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688459 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-dshm\") pod \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " Jul 15 23:12:57.688560 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-home\") pod \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\" (UID: \"89edbc6f-e2a5-4f02-a51f-cbcca05914b5\") " Jul 15 23:12:57.688825 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688673 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "89edbc6f-e2a5-4f02-a51f-cbcca05914b5" (UID: "89edbc6f-e2a5-4f02-a51f-cbcca05914b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:12:57.688825 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688695 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89edbc6f-e2a5-4f02-a51f-cbcca05914b5" (UID: "89edbc6f-e2a5-4f02-a51f-cbcca05914b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:12:57.688933 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688872 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:57.688933 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688884 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-home" (OuterVolumeSpecName: "home") pod "89edbc6f-e2a5-4f02-a51f-cbcca05914b5" (UID: "89edbc6f-e2a5-4f02-a51f-cbcca05914b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:12:57.688933 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.688889 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:57.691049 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.691012 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-dshm" (OuterVolumeSpecName: "dshm") pod "89edbc6f-e2a5-4f02-a51f-cbcca05914b5" (UID: "89edbc6f-e2a5-4f02-a51f-cbcca05914b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:12:57.691049 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.691030 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89edbc6f-e2a5-4f02-a51f-cbcca05914b5" (UID: "89edbc6f-e2a5-4f02-a51f-cbcca05914b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:12:57.691279 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.691150 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-kube-api-access-7hmwk" (OuterVolumeSpecName: "kube-api-access-7hmwk") pod "89edbc6f-e2a5-4f02-a51f-cbcca05914b5" (UID: "89edbc6f-e2a5-4f02-a51f-cbcca05914b5"). InnerVolumeSpecName "kube-api-access-7hmwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:12:57.727391 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.727353 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:12:57.789650 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.789613 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:57.789650 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.789646 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmwk\" (UniqueName: \"kubernetes.io/projected/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-kube-api-access-7hmwk\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:57.789859 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.789666 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:57.789859 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.789675 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89edbc6f-e2a5-4f02-a51f-cbcca05914b5-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:57.828478 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:57.828436 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:12:58.339501 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.339464 2565 generic.go:356] "Generic (PLEG): container finished" podID="89edbc6f-e2a5-4f02-a51f-cbcca05914b5" containerID="314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9" exitCode=0 Jul 15 23:12:58.339794 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.339507 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" Jul 15 23:12:58.339794 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.339516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" event={"ID":"89edbc6f-e2a5-4f02-a51f-cbcca05914b5","Type":"ContainerDied","Data":"314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9"} Jul 15 23:12:58.339794 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.339555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx" event={"ID":"89edbc6f-e2a5-4f02-a51f-cbcca05914b5","Type":"ContainerDied","Data":"0f84a75cec2339b864bf205a1ae48df24f63b7d7724f42cfc2d933f1c636b728"} Jul 15 23:12:58.339794 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.339577 2565 scope.go:117] "RemoveContainer" containerID="314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9" Jul 15 23:12:58.348654 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.348637 2565 scope.go:117] "RemoveContainer" containerID="314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9" Jul 15 23:12:58.348918 ip-10-0-130-194 kubenswrapper[2565]: E0715 23:12:58.348899 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9\": container with ID starting with 314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9 not found: ID does not exist" containerID="314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9" Jul 15 23:12:58.348956 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.348927 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9"} err="failed to get container status \"314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9\": rpc error: code = NotFound desc = could not find container \"314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9\": container with ID starting with 314563c814c96cfa840b44d27f39fec4ac99ebec1bfd6d23a5038ce52a8564b9 not found: ID does not exist" Jul 15 23:12:58.360037 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.360013 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx"] Jul 15 23:12:58.363981 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.363960 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ab95dd/rate-test-model-ab95dd-kserve-6f49c58875-c7qxx"] Jul 15 23:12:58.383070 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.383036 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" podUID="6d45311c-c505-4936-b997-c4aa5edbd8d6" containerName="main" containerID="cri-o://8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7" gracePeriod=45 Jul 15 23:12:58.636209 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.636183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:12:58.699524 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.699492 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-dshm\") pod \"6d45311c-c505-4936-b997-c4aa5edbd8d6\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " Jul 15 23:12:58.699716 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.699553 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fftq\" (UniqueName: \"kubernetes.io/projected/6d45311c-c505-4936-b997-c4aa5edbd8d6-kube-api-access-4fftq\") pod \"6d45311c-c505-4936-b997-c4aa5edbd8d6\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " Jul 15 23:12:58.699716 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.699576 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-home\") pod \"6d45311c-c505-4936-b997-c4aa5edbd8d6\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " Jul 15 23:12:58.699716 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.699643 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d45311c-c505-4936-b997-c4aa5edbd8d6-tls-certs\") pod \"6d45311c-c505-4936-b997-c4aa5edbd8d6\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " Jul 15 23:12:58.699716 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.699689 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-model-cache\") pod \"6d45311c-c505-4936-b997-c4aa5edbd8d6\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " Jul 15 23:12:58.699942 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.699723 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-tmp-dir\") pod \"6d45311c-c505-4936-b997-c4aa5edbd8d6\" (UID: \"6d45311c-c505-4936-b997-c4aa5edbd8d6\") " Jul 15 23:12:58.699942 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.699834 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-home" (OuterVolumeSpecName: "home") pod "6d45311c-c505-4936-b997-c4aa5edbd8d6" (UID: "6d45311c-c505-4936-b997-c4aa5edbd8d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:12:58.699942 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.699904 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d45311c-c505-4936-b997-c4aa5edbd8d6" (UID: "6d45311c-c505-4936-b997-c4aa5edbd8d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:12:58.700101 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.700011 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-home\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:58.700101 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.700024 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-model-cache\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:58.700101 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.700051 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d45311c-c505-4936-b997-c4aa5edbd8d6" (UID: "6d45311c-c505-4936-b997-c4aa5edbd8d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:12:58.701944 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.701907 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d45311c-c505-4936-b997-c4aa5edbd8d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d45311c-c505-4936-b997-c4aa5edbd8d6" (UID: "6d45311c-c505-4936-b997-c4aa5edbd8d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:12:58.702080 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.701966 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d45311c-c505-4936-b997-c4aa5edbd8d6-kube-api-access-4fftq" (OuterVolumeSpecName: "kube-api-access-4fftq") pod "6d45311c-c505-4936-b997-c4aa5edbd8d6" (UID: "6d45311c-c505-4936-b997-c4aa5edbd8d6"). InnerVolumeSpecName "kube-api-access-4fftq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:12:58.702080 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.702007 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-dshm" (OuterVolumeSpecName: "dshm") pod "6d45311c-c505-4936-b997-c4aa5edbd8d6" (UID: "6d45311c-c505-4936-b997-c4aa5edbd8d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:12:58.801365 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.801326 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-dshm\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:58.801365 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.801364 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fftq\" (UniqueName: \"kubernetes.io/projected/6d45311c-c505-4936-b997-c4aa5edbd8d6-kube-api-access-4fftq\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:58.801365 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.801376 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d45311c-c505-4936-b997-c4aa5edbd8d6-tls-certs\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:58.801616 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:58.801386 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d45311c-c505-4936-b997-c4aa5edbd8d6-tmp-dir\") on node \"ip-10-0-130-194.ec2.internal\" DevicePath \"\"" Jul 15 23:12:59.346033 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.345996 2565 generic.go:356] "Generic (PLEG): container finished" podID="6d45311c-c505-4936-b997-c4aa5edbd8d6" containerID="8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7" exitCode=0 Jul 15 23:12:59.346033 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.346037 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" Jul 15 23:12:59.346313 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.346081 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" event={"ID":"6d45311c-c505-4936-b997-c4aa5edbd8d6","Type":"ContainerDied","Data":"8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7"} Jul 15 23:12:59.346313 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.346116 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75" event={"ID":"6d45311c-c505-4936-b997-c4aa5edbd8d6","Type":"ContainerDied","Data":"dfd1698771bc7c890c59e5370f06d16af4355a57c200b367c1485867cf230363"} Jul 15 23:12:59.346313 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.346134 2565 scope.go:117] "RemoveContainer" containerID="8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7" Jul 15 23:12:59.355675 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.355659 2565 scope.go:117] "RemoveContainer" containerID="8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7" Jul 15 23:12:59.355955 ip-10-0-130-194 kubenswrapper[2565]: E0715 23:12:59.355939 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7\": container with ID starting with 8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7 not found: ID does not exist" containerID="8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7" Jul 15 23:12:59.356000 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.355964 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7"} err="failed to get container status \"8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7\": rpc error: code = NotFound desc = could not find container \"8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7\": container with ID starting with 8f3b5e4ba1d1801d098866567c13988659520554844a4993c00e150f8b98a8b7 not found: ID does not exist" Jul 15 23:12:59.366964 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.366931 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75"] Jul 15 23:12:59.370740 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.370711 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6dbd3/rate-test-model-c6dbd3-kserve-69d4447d67-krs75"] Jul 15 23:12:59.585065 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.585029 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d45311c-c505-4936-b997-c4aa5edbd8d6" path="/var/lib/kubelet/pods/6d45311c-c505-4936-b997-c4aa5edbd8d6/volumes" Jul 15 23:12:59.585458 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:12:59.585444 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89edbc6f-e2a5-4f02-a51f-cbcca05914b5" path="/var/lib/kubelet/pods/89edbc6f-e2a5-4f02-a51f-cbcca05914b5/volumes" Jul 15 23:13:02.639147 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:02.639097 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:08.567365 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:08.567317 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:10.215975 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:10.215941 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:10.317094 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:10.317036 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:12.951658 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:12.951622 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:13:15.205872 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:15.205806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:15.305779 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:15.305738 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:15.476191 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:15.476054 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:15.685930 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:15.685886 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:16.072483 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:16.072452 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56b9548c98-sm2cz_6ebe998d-d511-4e83-97a1-0bb900356251/authorino/0.log" Jul 15 23:13:21.225295 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:21.225264 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lk4z6_5c5fa0f1-2df0-449f-9b57-ca787b186d2d/manager/0.log" Jul 15 23:13:21.348432 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:21.348401 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8rkb7_36d6a42b-91be-41e8-9286-42e55349e31f/manager/0.log" Jul 15 23:13:21.575966 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:21.575886 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mzwkw_93b6e213-1194-4025-a533-d426f495a78d/server/0.log" Jul 15 23:13:21.689841 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:21.689808 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ffnjm_8f7f3924-3714-47c3-bab4-56a63584aaf7/manager/0.log" Jul 15 23:13:22.046459 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:22.046432 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7db6597b7c-rlbg4_7c747a35-2e31-4a36-9794-f32e4f59b81e/copy-manifests/0.log" Jul 15 23:13:22.057937 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:22.057912 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7db6597b7c-rlbg4_7c747a35-2e31-4a36-9794-f32e4f59b81e/manager/0.log" Jul 15 23:13:23.822471 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:23.822440 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-ait-5d538e1f-29735940-w5xdd_ac8a1671-124f-4a85-92be-e2e8b026c7b7/cleanup/0.log" Jul 15 23:13:25.146180 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:25.146144 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-brmnx"] Jul 15 23:13:25.272284 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:25.272252 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56b9548c98-sm2cz_6ebe998d-d511-4e83-97a1-0bb900356251/authorino/0.log" Jul 15 23:13:25.384387 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:25.384363 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qhb4j_be8d7941-2563-4761-ba43-d9835c34e994/manager/0.log" Jul 15 23:13:25.597150 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:25.597051 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-c7rxl_c246eba4-afd0-4cf9-b79f-4bf8415cb8fa/kuadrant-console-plugin/0.log" Jul 15 23:13:25.712083 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:25.712042 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wmh75_47708cb9-7dff-4044-97f0-125a8ba74428/registry-server/0.log" Jul 15 23:13:25.836260 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:25.836208 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bpgpz_2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab/manager/0.log" Jul 15 23:13:25.951102 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:25.951072 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-brmnx_d2195f4a-fa10-48fa-9813-bd40bf370ab4/limitador/0.log" Jul 15 23:13:26.063765 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:26.063738 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sz8bk_a5228b7e-f25e-41f8-8fb8-8cb982becec2/manager/0.log" Jul 15 23:13:26.741953 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:26.741919 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69cf9c9cdb-m84hk_ad7132bc-cceb-4ae8-a87f-66043bd3a762/kube-auth-proxy/0.log" Jul 15 23:13:27.876905 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:27.876849 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9whnk_bf74c3eb-eacb-4435-9b36-dea6921b5910/main/0.log" Jul 15 23:13:28.091361 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:28.091318 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-8cxkf_a02c8d76-8d18-4ff6-82d1-9a2bdf1457bc/main/0.log" Jul 15 23:13:33.511002 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.510916 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wh22x/must-gather-xcmx2"] Jul 15 23:13:33.511416 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.511267 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89edbc6f-e2a5-4f02-a51f-cbcca05914b5" containerName="main" Jul 15 23:13:33.511416 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.511278 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="89edbc6f-e2a5-4f02-a51f-cbcca05914b5" containerName="main" Jul 15 23:13:33.511416 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.511297 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d45311c-c505-4936-b997-c4aa5edbd8d6" containerName="main" Jul 15 23:13:33.511416 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.511302 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d45311c-c505-4936-b997-c4aa5edbd8d6" containerName="main" Jul 15 23:13:33.511416 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.511352 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d45311c-c505-4936-b997-c4aa5edbd8d6" containerName="main" Jul 15 23:13:33.511416 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.511360 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="89edbc6f-e2a5-4f02-a51f-cbcca05914b5" containerName="main" Jul 15 23:13:33.514555 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.514535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wh22x/must-gather-xcmx2" Jul 15 23:13:33.517466 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.517432 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wh22x\"/\"kube-root-ca.crt\"" Jul 15 23:13:33.517466 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.517446 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wh22x\"/\"default-dockercfg-77s8m\"" Jul 15 23:13:33.517681 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.517442 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wh22x\"/\"openshift-service-ca.crt\"" Jul 15 23:13:33.524775 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.524745 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wh22x/must-gather-xcmx2"] Jul 15 23:13:33.528154 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.528127 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:33.600633 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.600590 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" limit=1024 Jul 15 23:13:33.610779 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.610747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77ts5\" (UniqueName: \"kubernetes.io/projected/53fa08e0-e303-4081-a09d-4dab6e0c8b42-kube-api-access-77ts5\") pod \"must-gather-xcmx2\" (UID: \"53fa08e0-e303-4081-a09d-4dab6e0c8b42\") " pod="openshift-must-gather-wh22x/must-gather-xcmx2" Jul 15 23:13:33.610938 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.610787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53fa08e0-e303-4081-a09d-4dab6e0c8b42-must-gather-output\") pod \"must-gather-xcmx2\" (UID: \"53fa08e0-e303-4081-a09d-4dab6e0c8b42\") " pod="openshift-must-gather-wh22x/must-gather-xcmx2" Jul 15 23:13:33.712182 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.712131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77ts5\" (UniqueName: \"kubernetes.io/projected/53fa08e0-e303-4081-a09d-4dab6e0c8b42-kube-api-access-77ts5\") pod \"must-gather-xcmx2\" (UID: \"53fa08e0-e303-4081-a09d-4dab6e0c8b42\") " pod="openshift-must-gather-wh22x/must-gather-xcmx2" Jul 15 23:13:33.712358 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.712212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53fa08e0-e303-4081-a09d-4dab6e0c8b42-must-gather-output\") pod \"must-gather-xcmx2\" (UID: \"53fa08e0-e303-4081-a09d-4dab6e0c8b42\") " pod="openshift-must-gather-wh22x/must-gather-xcmx2" Jul 15 23:13:33.712574 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.712558 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53fa08e0-e303-4081-a09d-4dab6e0c8b42-must-gather-output\") pod \"must-gather-xcmx2\" (UID: \"53fa08e0-e303-4081-a09d-4dab6e0c8b42\") " pod="openshift-must-gather-wh22x/must-gather-xcmx2" Jul 15 23:13:33.721491 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.721455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77ts5\" (UniqueName: \"kubernetes.io/projected/53fa08e0-e303-4081-a09d-4dab6e0c8b42-kube-api-access-77ts5\") pod \"must-gather-xcmx2\" (UID: \"53fa08e0-e303-4081-a09d-4dab6e0c8b42\") " pod="openshift-must-gather-wh22x/must-gather-xcmx2" Jul 15 23:13:33.826491 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.826390 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wh22x/must-gather-xcmx2" Jul 15 23:13:33.957303 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.957275 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wh22x/must-gather-xcmx2"] Jul 15 23:13:33.959505 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:13:33.959478 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53fa08e0_e303_4081_a09d_4dab6e0c8b42.slice/crio-28fc8e1bdacf7ae6f13705ae9df6e0a0fb0cec99527fa656edadffcc683c0603 WatchSource:0}: Error finding container 28fc8e1bdacf7ae6f13705ae9df6e0a0fb0cec99527fa656edadffcc683c0603: Status 404 returned error can't find the container with id 28fc8e1bdacf7ae6f13705ae9df6e0a0fb0cec99527fa656edadffcc683c0603 Jul 15 23:13:33.961462 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:33.961442 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 23:13:34.530355 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:34.530317 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh22x/must-gather-xcmx2" event={"ID":"53fa08e0-e303-4081-a09d-4dab6e0c8b42","Type":"ContainerStarted","Data":"28fc8e1bdacf7ae6f13705ae9df6e0a0fb0cec99527fa656edadffcc683c0603"} Jul 15 23:13:35.539407 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:35.539365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh22x/must-gather-xcmx2" event={"ID":"53fa08e0-e303-4081-a09d-4dab6e0c8b42","Type":"ContainerStarted","Data":"f694c2d6833c88da39eaf293e916118ec26b0372d09a4550d2b15a42a25145e1"} Jul 15 23:13:35.539911 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:35.539421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh22x/must-gather-xcmx2" event={"ID":"53fa08e0-e303-4081-a09d-4dab6e0c8b42","Type":"ContainerStarted","Data":"def0c48bdfb0aad62cd8ef8cae60221c7f96635114b3237e256d5945507d9a44"} Jul 15 23:13:35.555904 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:35.555818 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wh22x/must-gather-xcmx2" podStartSLOduration=1.735646389 podStartE2EDuration="2.5557968s" totalImagesPullingTime="820.150411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:13:35.55454713 +0000 UTC m=+2426.494749681" watchObservedRunningTime="2026-07-15 23:13:35.5557968 +0000 UTC m=+2426.495999351" Jul 15 23:13:36.314637 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:36.314608 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gp62w_2a63fe02-5823-4b44-b017-059e1e8ab33d/global-pull-secret-syncer/0.log" Jul 15 23:13:36.401280 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:36.401221 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5kgf2_1bd15363-1a4e-4956-b662-f8c68f813bd9/konnectivity-agent/0.log" Jul 15 23:13:36.480440 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:36.480413 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-194.ec2.internal_433ef3a1b26ec88b78ef4885409e47ef/haproxy/0.log" Jul 15 23:13:40.903740 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:40.903648 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56b9548c98-sm2cz_6ebe998d-d511-4e83-97a1-0bb900356251/authorino/0.log" Jul 15 23:13:40.943915 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:40.943790 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qhb4j_be8d7941-2563-4761-ba43-d9835c34e994/manager/0.log" Jul 15 23:13:41.030457 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:41.030424 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-c7rxl_c246eba4-afd0-4cf9-b79f-4bf8415cb8fa/kuadrant-console-plugin/0.log" Jul 15 23:13:41.088037 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:41.088002 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wmh75_47708cb9-7dff-4044-97f0-125a8ba74428/registry-server/0.log" Jul 15 23:13:41.261007 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:41.260851 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bpgpz_2bfde4da-cf4f-4ea8-822b-bc7d5cc6beab/manager/0.log" Jul 15 23:13:41.289894 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:41.289864 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-brmnx_d2195f4a-fa10-48fa-9813-bd40bf370ab4/limitador/0.log" Jul 15 23:13:41.368514 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:41.368453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sz8bk_a5228b7e-f25e-41f8-8fb8-8cb982becec2/manager/0.log" Jul 15 23:13:43.388903 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:43.388863 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zc8w9_c668433e-ff6c-4321-82e4-a26430446e1b/node-exporter/0.log" Jul 15 23:13:43.409159 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:43.409077 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zc8w9_c668433e-ff6c-4321-82e4-a26430446e1b/kube-rbac-proxy/0.log" Jul 15 23:13:43.429949 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:43.429913 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zc8w9_c668433e-ff6c-4321-82e4-a26430446e1b/init-textfile/0.log" Jul 15 23:13:48.566450 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:48.566420 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rsk5w_bd6596a2-25e0-4482-bd65-38fb27ddfe8d/dns/0.log" Jul 15 23:13:48.588039 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:48.588012 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rsk5w_bd6596a2-25e0-4482-bd65-38fb27ddfe8d/kube-rbac-proxy/0.log" Jul 15 23:13:48.633222 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:48.633189 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hdrnr_1cf1c5d5-ba93-4ef4-b043-8233b97b14d3/dns-node-resolver/0.log" Jul 15 23:13:49.173807 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:49.173779 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-845987996-slz86_f7743a76-2485-4a98-9c05-945ce06ca4d1/registry/0.log" Jul 15 23:13:49.215960 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:49.215928 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jfgkz_5d432250-ab22-419b-b783-26186a2bcfbf/node-ca/0.log" Jul 15 23:13:50.302778 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.302747 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69cf9c9cdb-m84hk_ad7132bc-cceb-4ae8-a87f-66043bd3a762/kube-auth-proxy/0.log" Jul 15 23:13:50.397481 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.397433 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f"] Jul 15 23:13:50.406242 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.406210 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.407956 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.407915 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f"] Jul 15 23:13:50.493658 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.493618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-lib-modules\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.493947 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.493908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-proc\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.494089 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.494074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-podres\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.494230 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.494212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhrq\" (UniqueName: \"kubernetes.io/projected/9db765c2-bf13-4984-9baf-9aa57c24a350-kube-api-access-vjhrq\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.494392 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.494375 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-sys\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.595725 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.595619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-lib-modules\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.595725 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.595664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-proc\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.595964 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.595740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-proc\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.595964 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.595769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-podres\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.595964 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.595818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhrq\" (UniqueName: \"kubernetes.io/projected/9db765c2-bf13-4984-9baf-9aa57c24a350-kube-api-access-vjhrq\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.595964 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.595839 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-lib-modules\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.595964 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.595899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-sys\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.595964 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.595926 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-podres\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.596216 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.596038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9db765c2-bf13-4984-9baf-9aa57c24a350-sys\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.603877 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.603848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhrq\" (UniqueName: \"kubernetes.io/projected/9db765c2-bf13-4984-9baf-9aa57c24a350-kube-api-access-vjhrq\") pod \"perf-node-gather-daemonset-bmp5f\" (UID: \"9db765c2-bf13-4984-9baf-9aa57c24a350\") " pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.718636 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.718603 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:50.892769 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:50.892742 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f"] Jul 15 23:13:50.899922 ip-10-0-130-194 kubenswrapper[2565]: W0715 23:13:50.899883 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9db765c2_bf13_4984_9baf_9aa57c24a350.slice/crio-0c43a2af56e6faa50a3dc32e5d79e7d336912a67bdf40b366a69b6f3f8bde8b2 WatchSource:0}: Error finding container 0c43a2af56e6faa50a3dc32e5d79e7d336912a67bdf40b366a69b6f3f8bde8b2: Status 404 returned error can't find the container with id 0c43a2af56e6faa50a3dc32e5d79e7d336912a67bdf40b366a69b6f3f8bde8b2 Jul 15 23:13:51.158265 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:51.158152 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h77p9_acf3b5e8-bdb1-4831-a9b3-6febe54464cd/serve-healthcheck-canary/0.log" Jul 15 23:13:51.646375 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:51.646262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" event={"ID":"9db765c2-bf13-4984-9baf-9aa57c24a350","Type":"ContainerStarted","Data":"bbdeb8b6f158f057ff19f94165698c3ed302c16e8c8860de9522f0fed8eb6268"} Jul 15 23:13:51.646375 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:51.646336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" event={"ID":"9db765c2-bf13-4984-9baf-9aa57c24a350","Type":"ContainerStarted","Data":"0c43a2af56e6faa50a3dc32e5d79e7d336912a67bdf40b366a69b6f3f8bde8b2"} Jul 15 23:13:51.647058 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:51.647039 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:13:51.665194 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:51.665128 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" podStartSLOduration=1.665106714 podStartE2EDuration="1.665106714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:13:51.663468506 +0000 UTC m=+2442.603671057" watchObservedRunningTime="2026-07-15 23:13:51.665106714 +0000 UTC m=+2442.605309264" Jul 15 23:13:51.818295 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:51.818266 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4tsd_9bfad52c-9805-44c2-b8e0-5ae8243987d0/kube-rbac-proxy/0.log" Jul 15 23:13:51.837434 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:51.837400 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4tsd_9bfad52c-9805-44c2-b8e0-5ae8243987d0/exporter/0.log" Jul 15 23:13:51.857589 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:51.857561 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4tsd_9bfad52c-9805-44c2-b8e0-5ae8243987d0/extractor/0.log" Jul 15 23:13:53.943984 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:53.943958 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lk4z6_5c5fa0f1-2df0-449f-9b57-ca787b186d2d/manager/0.log" Jul 15 23:13:54.086144 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:54.086108 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8rkb7_36d6a42b-91be-41e8-9286-42e55349e31f/manager/0.log" Jul 15 23:13:54.192603 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:54.192574 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mzwkw_93b6e213-1194-4025-a533-d426f495a78d/server/0.log" Jul 15 23:13:54.265569 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:54.265484 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ffnjm_8f7f3924-3714-47c3-bab4-56a63584aaf7/manager/0.log" Jul 15 23:13:54.442292 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:54.442256 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7db6597b7c-rlbg4_7c747a35-2e31-4a36-9794-f32e4f59b81e/manager/0.log" Jul 15 23:13:54.458702 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:54.458678 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7db6597b7c-rlbg4_7c747a35-2e31-4a36-9794-f32e4f59b81e/copy-manifests/0.log" Jul 15 23:13:55.648247 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:55.648211 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-24dtx_1a5e4284-d779-4938-b80f-60e59c807fc9/openshift-lws-operator/0.log" Jul 15 23:13:56.893443 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:56.893406 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69cf9c9cdb-m84hk_ad7132bc-cceb-4ae8-a87f-66043bd3a762/kube-auth-proxy/0.log" Jul 15 23:13:58.208534 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:58.208500 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69cf9c9cdb-m84hk_ad7132bc-cceb-4ae8-a87f-66043bd3a762/kube-auth-proxy/0.log" Jul 15 23:13:58.669486 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:13:58.669452 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wh22x/perf-node-gather-daemonset-bmp5f" Jul 15 23:14:00.530782 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:00.530701 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-hrns4_0225982a-bd96-4dff-b3f3-a960b4a53d14/migrator/0.log" Jul 15 23:14:00.552825 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:00.552789 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-hrns4_0225982a-bd96-4dff-b3f3-a960b4a53d14/graceful-termination/0.log" Jul 15 23:14:02.111436 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.111346 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bx82_a449bf70-dfdc-4c4a-9248-ae820d7dbe71/kube-multus-additional-cni-plugins/0.log" Jul 15 23:14:02.132189 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.132127 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bx82_a449bf70-dfdc-4c4a-9248-ae820d7dbe71/egress-router-binary-copy/0.log" Jul 15 23:14:02.151954 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.151928 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bx82_a449bf70-dfdc-4c4a-9248-ae820d7dbe71/cni-plugins/0.log" Jul 15 23:14:02.171953 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.171921 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bx82_a449bf70-dfdc-4c4a-9248-ae820d7dbe71/bond-cni-plugin/0.log" Jul 15 23:14:02.191732 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.191698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bx82_a449bf70-dfdc-4c4a-9248-ae820d7dbe71/routeoverride-cni/0.log" Jul 15 23:14:02.210473 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.210443 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bx82_a449bf70-dfdc-4c4a-9248-ae820d7dbe71/whereabouts-cni-bincopy/0.log" Jul 15 23:14:02.229041 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.229011 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6bx82_a449bf70-dfdc-4c4a-9248-ae820d7dbe71/whereabouts-cni/0.log" Jul 15 23:14:02.592077 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.592040 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pk7xr_27f508ca-8bf5-4bfb-9e27-4f543593132c/kube-multus/0.log" Jul 15 23:14:02.694030 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.693999 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jhmzm_7b9b6737-f701-4ecb-86f1-0c25f73efdf6/network-metrics-daemon/0.log" Jul 15 23:14:02.721526 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:02.721500 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jhmzm_7b9b6737-f701-4ecb-86f1-0c25f73efdf6/kube-rbac-proxy/0.log" Jul 15 23:14:03.991487 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:03.991454 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qj5gc_7c4945d4-04f4-480b-9abb-5db6c5a8a87c/ovn-controller/0.log" Jul 15 23:14:04.007255 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:04.007223 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qj5gc_7c4945d4-04f4-480b-9abb-5db6c5a8a87c/ovn-acl-logging/0.log" Jul 15 23:14:04.028561 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:04.028532 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qj5gc_7c4945d4-04f4-480b-9abb-5db6c5a8a87c/kube-rbac-proxy-node/0.log" Jul 15 23:14:04.049429 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:04.049394 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qj5gc_7c4945d4-04f4-480b-9abb-5db6c5a8a87c/kube-rbac-proxy-ovn-metrics/0.log" Jul 15 23:14:04.066984 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:04.066960 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qj5gc_7c4945d4-04f4-480b-9abb-5db6c5a8a87c/northd/0.log" Jul 15 23:14:04.086434 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:04.086411 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qj5gc_7c4945d4-04f4-480b-9abb-5db6c5a8a87c/nbdb/0.log" Jul 15 23:14:04.108267 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:04.108241 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qj5gc_7c4945d4-04f4-480b-9abb-5db6c5a8a87c/sbdb/0.log" Jul 15 23:14:04.160064 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:04.160035 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qj5gc_7c4945d4-04f4-480b-9abb-5db6c5a8a87c/ovnkube-controller/0.log" Jul 15 23:14:05.493491 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:05.493451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-v26ns_a3257827-435d-40d8-ad19-955d7041302e/network-check-target-container/0.log" Jul 15 23:14:06.539596 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:06.539571 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6vt2l_2c753c4f-e47f-4f3f-8fe2-4d41fa627ea9/iptables-alerter/0.log" Jul 15 23:14:07.272274 ip-10-0-130-194 kubenswrapper[2565]: I0715 23:14:07.272244 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7xskg_1daa7794-3d7e-43b0-b053-6ff4d17f71ec/tuned/0.log"