Jul 07 15:40:29.027094 ip-10-0-137-175 systemd[1]: Starting Kubernetes Kubelet... Jul 07 15:40:29.348154 ip-10-0-137-175 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:40:29.348154 ip-10-0-137-175 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 15:40:29.348154 ip-10-0-137-175 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:40:29.348154 ip-10-0-137-175 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 15:40:29.348154 ip-10-0-137-175 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:40:29.348905 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.348822 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 15:40:29.352814 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352798 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:29.352814 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352814 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352818 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352821 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352826 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352830 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352834 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352837 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352841 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352845 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352849 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352852 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352855 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352858 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352860 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352864 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352867 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352870 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352872 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352875 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:29.352877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352878 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352880 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352883 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352886 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352889 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352892 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352895 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352897 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352900 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352903 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352906 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352908 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352911 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352913 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352916 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352918 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352921 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352923 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352926 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352928 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:29.353320 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352930 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352933 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352935 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352938 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352940 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352943 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352945 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352948 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352951 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352954 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352956 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352959 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352961 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352963 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352966 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352969 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352972 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352974 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352977 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:29.353859 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352979 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352982 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352984 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352987 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352989 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352992 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352995 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.352998 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353002 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353005 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353008 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353010 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353013 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353016 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353018 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353020 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353023 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353025 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353028 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:29.354323 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353030 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353033 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353035 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353038 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353040 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353043 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353045 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353048 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353051 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353053 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353055 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353058 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353061 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353063 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353066 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353068 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353071 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353073 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353076 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353078 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:29.354772 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.353081 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354940 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354949 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354955 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354958 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354961 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354964 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354967 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354970 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354973 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354976 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354979 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354982 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354984 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354987 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354990 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354993 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354995 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.354998 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:29.355265 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355000 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355003 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355005 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355008 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355010 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355013 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355017 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355019 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355021 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355024 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355026 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355029 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355031 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355034 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355036 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355040 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355043 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355045 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355048 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355050 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:29.355822 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355053 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355056 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355058 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355061 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355063 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355065 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355068 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355070 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355073 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355077 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355080 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355083 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355086 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355088 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355091 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355093 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355096 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355098 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355101 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355103 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:29.356407 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355106 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355110 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355113 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355116 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355119 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355123 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355126 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355129 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355132 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355135 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355137 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355140 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355142 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355144 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355147 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355149 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355152 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355154 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:29.357143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355157 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355159 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355161 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355164 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355166 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355170 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355173 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355175 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355178 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355180 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355183 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355185 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355187 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355190 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355192 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355195 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355197 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355200 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355202 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355204 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:29.357583 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355207 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355210 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.355213 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357011 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357025 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357031 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357035 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357038 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357045 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357050 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357168 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357171 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357174 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357177 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357180 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357185 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357188 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357191 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357193 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:29.358100 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357196 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357198 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357201 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357204 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357206 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357209 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357212 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357214 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357217 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357219 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357222 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357225 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357228 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357231 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357233 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357235 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357238 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357241 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357244 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357247 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:29.358560 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357249 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357252 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357254 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357256 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357259 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357261 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357264 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357266 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357268 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357271 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357273 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357276 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357280 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357283 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357286 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357288 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357291 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357293 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357296 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357298 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:29.359053 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357301 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357303 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357306 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357309 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357312 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357314 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357317 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357319 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357322 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357325 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357328 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357331 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357333 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357336 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357338 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357341 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357343 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357346 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357348 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357351 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:29.359569 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357353 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357356 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357358 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357360 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357363 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357365 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357368 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357371 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357373 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357376 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357378 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357381 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357384 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357388 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357390 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357393 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357396 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357400 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357402 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:29.360072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357405 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357410 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357415 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.357418 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357496 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357503 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357511 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357515 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357519 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357522 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357527 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357531 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357534 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357536 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357540 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357543 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357546 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357549 2576 flags.go:64] FLAG: --cgroup-root="" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357552 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357554 2576 flags.go:64] FLAG: --client-ca-file="" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357557 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357560 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357565 2576 flags.go:64] FLAG: --cluster-domain="" Jul 07 15:40:29.360518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357568 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357573 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357576 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357579 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357583 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357587 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357590 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357593 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357596 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357599 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357602 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357605 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357608 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357613 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357616 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357618 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357621 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357624 2576 flags.go:64] FLAG: --enable-server="true" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357627 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357631 2576 flags.go:64] FLAG: --event-burst="100" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357634 2576 flags.go:64] FLAG: --event-qps="50" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357637 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357640 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357643 2576 flags.go:64] FLAG: --eviction-hard="" Jul 07 15:40:29.361108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357646 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357649 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357652 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357655 2576 flags.go:64] FLAG: --eviction-soft="" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357658 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357661 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357663 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357667 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357669 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357674 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357676 2576 flags.go:64] FLAG: --feature-gates="" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357680 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357683 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357685 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357689 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357691 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357694 2576 flags.go:64] FLAG: --help="false" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357697 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357700 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357703 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357706 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357709 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357713 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357715 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 15:40:29.361676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357718 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357721 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357723 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357726 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357729 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357731 2576 flags.go:64] FLAG: --kube-reserved="" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357734 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357737 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357739 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357742 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357745 2576 flags.go:64] FLAG: --lock-file="" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357747 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357750 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357752 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357757 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357760 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357764 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357768 2576 flags.go:64] FLAG: --logging-format="text" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357771 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357789 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357793 2576 flags.go:64] FLAG: --manifest-url="" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357795 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357800 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357802 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357806 2576 flags.go:64] FLAG: --max-pods="110" Jul 07 15:40:29.362282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357809 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357812 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357816 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357819 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357822 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357825 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357828 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357836 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357839 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357842 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357845 2576 flags.go:64] FLAG: --pod-cidr="" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357848 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357853 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357856 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357859 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357862 2576 flags.go:64] FLAG: --port="10250" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357865 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357867 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-079fc5137e670eef5" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357870 2576 flags.go:64] FLAG: --qos-reserved="" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357873 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357876 2576 flags.go:64] FLAG: --register-node="true" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357879 2576 flags.go:64] FLAG: --register-with-taints="" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357882 2576 flags.go:64] FLAG: --registry-burst="10" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357885 2576 flags.go:64] FLAG: --registry-qps="5" Jul 07 15:40:29.362900 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357890 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357892 2576 flags.go:64] FLAG: --reserved-memory="" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357896 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357899 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357902 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357904 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357907 2576 flags.go:64] FLAG: --runonce="false" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357910 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357913 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357916 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357918 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357921 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357924 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357927 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357929 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357932 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357935 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357937 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357940 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357943 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357946 2576 flags.go:64] FLAG: --system-cgroups="" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357948 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357954 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357957 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357960 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 15:40:29.363482 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357964 2576 flags.go:64] FLAG: --tls-min-version="" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357966 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357969 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357972 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357975 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357978 2576 flags.go:64] FLAG: --v="2" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357981 2576 flags.go:64] FLAG: --version="false" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357987 2576 flags.go:64] FLAG: --vmodule="" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357991 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.357994 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358103 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358108 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358111 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358114 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358117 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358120 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358123 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358125 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358128 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358130 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358133 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358135 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:29.364117 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358138 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358140 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358143 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358145 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358147 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358150 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358153 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358155 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358157 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358160 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358162 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358164 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358167 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358169 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358171 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358174 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358176 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358179 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358182 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358185 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:29.364678 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358187 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358201 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358204 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358207 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358213 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358216 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358218 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358221 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358224 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358226 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358229 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358232 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358235 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358237 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358241 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358244 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358247 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358250 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358253 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:29.365179 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358255 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358257 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358260 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358263 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358265 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358268 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358270 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358272 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358275 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358278 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358280 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358283 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358285 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358288 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358290 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358293 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358295 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358299 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358301 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358304 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:29.365639 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358306 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358309 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358311 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358314 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358316 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358318 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358321 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358323 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358326 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358328 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358332 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358335 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358337 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358340 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358344 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358347 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358350 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358352 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358354 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:29.366149 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358357 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358360 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358362 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358364 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358367 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358369 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358372 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358375 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.358377 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.358382 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.365517 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.365533 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.365552 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.365558 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 15:40:29.366598 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365610 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365617 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365621 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365624 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365627 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365630 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365633 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365635 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365638 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365641 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365644 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365648 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365650 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365653 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365656 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365659 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365661 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365664 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365666 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365668 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:29.367023 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365671 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365673 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365676 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365678 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365680 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365683 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365685 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365688 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365690 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365695 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365698 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365701 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365703 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365705 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365708 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365710 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365712 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365715 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365717 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:29.367502 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365719 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365722 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365724 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365726 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365729 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365731 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365734 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365737 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365739 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365742 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365744 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365746 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365749 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365751 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365753 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365756 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365758 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365761 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365764 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365767 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:29.367973 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365771 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365791 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365795 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365798 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365802 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365804 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365807 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365810 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365812 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365815 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365817 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365820 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365822 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365825 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365829 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365832 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365835 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365837 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365840 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365842 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:29.368445 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365845 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365847 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365850 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365852 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365855 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365857 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365860 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365862 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365865 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365867 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365869 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365872 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365874 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365877 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365880 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365883 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365886 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365888 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365890 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:29.369213 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.365893 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.365898 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366022 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366026 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366029 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366032 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366035 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366037 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366039 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366042 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366044 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366047 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366049 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366052 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366054 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:40:29.369683 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366057 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366059 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366062 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366064 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366066 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366069 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366071 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366073 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366076 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366078 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366081 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366085 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366088 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366091 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366094 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366099 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366101 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366104 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366106 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366109 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:40:29.370075 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366111 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366113 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366116 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366118 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366120 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366123 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366125 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366128 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366130 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366132 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366135 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366137 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366140 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366142 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366145 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366147 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366150 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366152 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366155 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366158 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:40:29.370614 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366160 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366163 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366166 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366168 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366171 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366173 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366176 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366178 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366182 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366185 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366188 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366192 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366196 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366199 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366201 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366205 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366207 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366210 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366212 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:40:29.371242 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366215 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366218 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366220 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366223 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366226 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366229 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366231 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366234 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366236 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366239 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366241 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366244 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366246 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366248 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366251 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366253 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366256 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366258 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366262 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366265 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:40:29.372018 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366268 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366270 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366273 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366275 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366278 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366280 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:29.366283 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.366287 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.366390 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.369023 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.370185 2576 server.go:1019] "Starting client certificate rotation" Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.370304 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.370908 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 15:40:29.372611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.372394 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 15:40:29.373579 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.373564 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 15:40:29.382615 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.382600 2576 log.go:25] "Validated CRI v1 runtime API" Jul 07 15:40:29.385946 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.385934 2576 log.go:25] "Validated CRI v1 image API" Jul 07 15:40:29.386771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.386752 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 15:40:29.391231 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.391213 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8d15cf96-b490-458a-8998-fe123289caaa:/dev/nvme0n1p3 9f3c61a7-1a5f-4bd3-a5cb-cd62e8f10108:/dev/nvme0n1p4] Jul 07 15:40:29.391296 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.391231 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 15:40:29.395135 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.395114 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 15:40:29.397370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.397266 2576 manager.go:216] Machine: {Timestamp:2026-07-07 15:40:29.396081154 +0000 UTC m=+0.267341332 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102530 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c8abf1a5ee3776527600995749cf2 SystemUUID:ec2c8abf-1a5e-e377-6527-600995749cf2 BootID:463934b1-f36f-4e35-a705-f7925c4e4239 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:cc:81:1c:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:cc:81:1c:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:7a:a2:1a:dc:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 15:40:29.397370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.397362 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 15:40:29.397481 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.397440 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 15:40:29.400005 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.399982 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 15:40:29.400133 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.400008 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-175.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 15:40:29.400644 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.400635 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 15:40:29.400677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.400647 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 15:40:29.400677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.400655 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 15:40:29.401183 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.401173 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 15:40:29.402569 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.402560 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 15:40:29.406143 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406131 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 07 15:40:29.406255 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406245 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 15:40:29.406294 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406276 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 07 15:40:29.406294 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406293 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 15:40:29.406347 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406308 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 15:40:29.406347 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406317 2576 kubelet.go:398] "Adding apiserver pod source" Jul 07 15:40:29.406347 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406325 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 15:40:29.406655 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406644 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 15:40:29.406702 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.406662 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 15:40:29.408404 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.408384 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 15:40:29.410242 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.410227 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 15:40:29.410291 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.410272 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 15:40:29.411544 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411531 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 15:40:29.411576 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411553 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 15:40:29.411576 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411563 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 15:40:29.411576 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411572 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 15:40:29.411677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411583 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 15:40:29.411677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411589 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 15:40:29.411677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411597 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 15:40:29.411677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411608 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 15:40:29.411677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411616 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 15:40:29.411677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411622 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 15:40:29.411677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411634 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 15:40:29.411677 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.411639 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 15:40:29.412872 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.412855 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 15:40:29.412872 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.412868 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 15:40:29.416158 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.416144 2576 server.go:1268] "Started kubelet" Jul 07 15:40:29.416306 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.416249 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 15:40:29.416371 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.416330 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 15:40:29.416432 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.416367 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 15:40:29.416911 ip-10-0-137-175 systemd[1]: Started Kubernetes Kubelet. Jul 07 15:40:29.417734 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.417589 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-175.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 15:40:29.417734 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.417628 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 15:40:29.417734 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.417698 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 15:40:29.418094 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.417916 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 15:40:29.418955 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.418941 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 07 15:40:29.421019 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.420287 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-175.ec2.internal.18c00c2a91fc9396 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-175.ec2.internal,UID:ip-10-0-137-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-175.ec2.internal,},FirstTimestamp:2026-07-07 15:40:29.416125334 +0000 UTC m=+0.287385491,LastTimestamp:2026-07-07 15:40:29.416125334 +0000 UTC m=+0.287385491,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-175.ec2.internal,}" Jul 07 15:40:29.421384 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.421366 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 15:40:29.421654 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.421640 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hqhdf" Jul 07 15:40:29.424870 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.424852 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 15:40:29.424870 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.424856 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 15:40:29.425170 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.425150 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:29.425589 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.425571 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 15:40:29.425589 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.425591 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 15:40:29.425690 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.425574 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 15:40:29.425722 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.425709 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 07 15:40:29.425722 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.425720 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 07 15:40:29.427008 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.426981 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 15:40:29.427324 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.427224 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 15:40:29.427579 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.427561 2576 factory.go:55] Registering systemd factory Jul 07 15:40:29.427646 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.427582 2576 factory.go:223] Registration of the systemd container factory successfully Jul 07 15:40:29.427859 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.427841 2576 factory.go:153] Registering CRI-O factory Jul 07 15:40:29.427971 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.427870 2576 factory.go:223] Registration of the crio container factory successfully Jul 07 15:40:29.427971 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.427922 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 15:40:29.427971 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.427945 2576 factory.go:103] Registering Raw factory Jul 07 15:40:29.427971 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.427961 2576 manager.go:1196] Started watching for new ooms in manager Jul 07 15:40:29.428556 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.428539 2576 manager.go:349] Starting recovery of all containers Jul 07 15:40:29.430425 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.430403 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hqhdf" Jul 07 15:40:29.439473 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.439372 2576 manager.go:354] Recovery completed Jul 07 15:40:29.443649 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.443637 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:29.445811 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.445796 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:29.445873 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.445824 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:29.445873 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.445833 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:29.446367 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.446352 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 15:40:29.446367 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.446366 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 15:40:29.446515 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.446386 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 07 15:40:29.446515 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.446397 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:29.448126 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.448111 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:29.448200 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.448137 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:29.448200 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.448146 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:29.448446 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.448434 2576 policy_none.go:49] "None policy: Start" Jul 07 15:40:29.448485 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.448451 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 15:40:29.448485 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.448462 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 15:40:29.450423 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.450411 2576 policy_none.go:47] "Start" Jul 07 15:40:29.487969 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.487955 2576 manager.go:337] "Starting Device Plugin manager" Jul 07 15:40:29.488059 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.487992 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 15:40:29.488059 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.488005 2576 server.go:87] "Starting device plugin registration server" Jul 07 15:40:29.488248 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.488235 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 15:40:29.488310 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.488249 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 15:40:29.488355 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.488336 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 15:40:29.488912 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.488897 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 15:40:29.488912 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.488910 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 15:40:29.489051 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.489008 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 15:40:29.489084 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.489061 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:29.526947 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.526912 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 15:40:29.533760 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.528195 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 15:40:29.533760 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.528227 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 15:40:29.533760 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.528247 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 15:40:29.533760 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.528275 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 15:40:29.533760 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.530356 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:29.588354 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.588324 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:29.589214 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.589198 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:29.589290 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.589230 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:29.589290 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.589246 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:29.589290 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.589269 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.596814 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.596801 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.596854 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.596822 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-175.ec2.internal\": node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:29.608849 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.608798 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:29.629200 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.629154 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal"] Jul 07 15:40:29.629294 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.629232 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:29.630032 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.630016 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:29.630103 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.630046 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:29.630103 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.630057 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:29.632417 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.632404 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:29.632549 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.632535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.632605 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.632569 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:29.633107 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.633092 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:29.633194 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.633117 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:29.633194 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.633126 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:29.633194 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.633097 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:29.633194 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.633191 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:29.633343 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.633201 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:29.635496 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.635482 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.635562 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.635507 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:40:29.636132 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.636117 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:40:29.636204 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.636143 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:40:29.636204 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.636159 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:40:29.660310 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.660293 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-175.ec2.internal\" not found" node="ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.664534 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.664521 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-175.ec2.internal\" not found" node="ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.709328 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.709304 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:29.726838 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.726810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c9f71f5b36a02e94770dabb884fd998-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal\" (UID: \"8c9f71f5b36a02e94770dabb884fd998\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.726921 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.726845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c9f71f5b36a02e94770dabb884fd998-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal\" (UID: \"8c9f71f5b36a02e94770dabb884fd998\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.726921 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.726874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed097c754df8ae0ea286f9bc978e4472-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.ec2.internal\" (UID: \"ed097c754df8ae0ea286f9bc978e4472\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.809689 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.809663 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:29.828025 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.827999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c9f71f5b36a02e94770dabb884fd998-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal\" (UID: \"8c9f71f5b36a02e94770dabb884fd998\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.828118 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.828032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c9f71f5b36a02e94770dabb884fd998-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal\" (UID: \"8c9f71f5b36a02e94770dabb884fd998\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.828118 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.828055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed097c754df8ae0ea286f9bc978e4472-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.ec2.internal\" (UID: \"ed097c754df8ae0ea286f9bc978e4472\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.828118 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.828099 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c9f71f5b36a02e94770dabb884fd998-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal\" (UID: \"8c9f71f5b36a02e94770dabb884fd998\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.828118 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.828108 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c9f71f5b36a02e94770dabb884fd998-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal\" (UID: \"8c9f71f5b36a02e94770dabb884fd998\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.828260 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.828108 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed097c754df8ae0ea286f9bc978e4472-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.ec2.internal\" (UID: \"ed097c754df8ae0ea286f9bc978e4472\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.910491 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:29.910422 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:29.962890 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.962865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:29.968005 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:29.967736 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" Jul 07 15:40:30.011464 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:30.011437 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:30.112047 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:30.112003 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:30.212636 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:30.212561 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:30.313099 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:30.313062 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:30.369602 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:30.369574 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 15:40:30.370108 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:30.369710 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 15:40:30.414038 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:30.414013 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:30.433924 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:30.433891 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 15:35:30 +0000 UTC" deadline="2028-03-23 20:51:57.800713692 +0000 UTC" Jul 07 15:40:30.433924 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:30.433917 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15005h11m27.366799429s" Jul 07 15:40:30.514189 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:30.514127 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:30.564856 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:30.564825 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded097c754df8ae0ea286f9bc978e4472.slice/crio-e2fbfd2969973a85e1eb1375b505bf7a70bd3aba3b47c8851615fd022e0747bf WatchSource:0}: Error finding container e2fbfd2969973a85e1eb1375b505bf7a70bd3aba3b47c8851615fd022e0747bf: Status 404 returned error can't find the container with id e2fbfd2969973a85e1eb1375b505bf7a70bd3aba3b47c8851615fd022e0747bf Jul 07 15:40:30.565074 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:30.565061 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9f71f5b36a02e94770dabb884fd998.slice/crio-543f8a393e65b95502462301b9fcc19aabc4c9170b25f01ab39f9918d5729db9 WatchSource:0}: Error finding container 543f8a393e65b95502462301b9fcc19aabc4c9170b25f01ab39f9918d5729db9: Status 404 returned error can't find the container with id 543f8a393e65b95502462301b9fcc19aabc4c9170b25f01ab39f9918d5729db9 Jul 07 15:40:30.569858 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:30.569841 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:40:31.887362 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:31.887327 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:31.934605 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:31.934584 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:31.988016 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:31.987964 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-175.ec2.internal\" not found" Jul 07 15:40:32.015066 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.015044 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:32.034055 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.034035 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:32.097865 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.097799 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" Jul 07 15:40:32.108843 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.108827 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 15:40:32.109676 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.109664 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" Jul 07 15:40:32.118198 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.118182 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 15:40:32.513381 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.513298 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:40:32.679829 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.679603 2576 apiserver.go:52] "Watching apiserver" Jul 07 15:40:32.688273 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.688243 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 15:40:32.689553 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.689511 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-ltsg9","openshift-network-operator/iptables-alerter-98ghd","openshift-ovn-kubernetes/ovnkube-node-8p85w","kube-system/global-pull-secret-syncer-cvlj7","kube-system/konnectivity-agent-xfwf2","kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw","openshift-dns/node-resolver-mjdkq","openshift-image-registry/node-ca-9grq8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal","openshift-multus/multus-additional-cni-plugins-44mlq","openshift-cluster-node-tuning-operator/tuned-2ll4t","openshift-multus/multus-22l6f","openshift-multus/network-metrics-daemon-27cjl"] Jul 07 15:40:32.692021 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.691988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.694512 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.694217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.695340 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.694747 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 15:40:32.695340 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.695081 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 15:40:32.695340 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.695140 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 15:40:32.695340 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.695157 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 15:40:32.695340 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.695267 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 15:40:32.695340 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.695295 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p5w4m\"" Jul 07 15:40:32.696808 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.696770 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:32.698104 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.698086 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 15:40:32.698768 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.698752 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:32.703370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.699367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.703370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.701839 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 15:40:32.703370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.701943 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g5qgl\"" Jul 07 15:40:32.703370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.702465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:32.703370 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.702537 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:32.703370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.703124 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 15:40:32.703370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.703231 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f67x4\"" Jul 07 15:40:32.703370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.703280 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 15:40:32.704214 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.703895 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 15:40:32.705376 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.705356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:32.705943 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.705923 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 15:40:32.706040 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.705986 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 15:40:32.706129 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.706115 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 15:40:32.707359 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.707339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.708217 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.708153 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 15:40:32.708441 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.708427 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rtzwd\"" Jul 07 15:40:32.708653 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.708637 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 15:40:32.710633 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kzc4f\"" Jul 07 15:40:32.710633 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710182 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.710633 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710183 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 15:40:32.710815 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-system-cni-dir\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.710815 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-slash\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.710815 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-etc-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.710815 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.710815 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbpw\" (UniqueName: \"kubernetes.io/projected/b1dbb731-10bf-478b-bc95-e162549bddd5-kube-api-access-jkbpw\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711071 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.711071 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0afbfec3-a252-483c-9b13-9ca1144a1eaf-host-slash\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.711071 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.710988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.711071 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711021 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fx58\" (UniqueName: \"kubernetes.io/projected/0afbfec3-a252-483c-9b13-9ca1144a1eaf-kube-api-access-5fx58\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.711258 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-ovn\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711258 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/791f6cb8-9518-45c6-aad0-aa14c4271724-agent-certs\") pod \"konnectivity-agent-xfwf2\" (UID: \"791f6cb8-9518-45c6-aad0-aa14c4271724\") " pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:32.711258 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/791f6cb8-9518-45c6-aad0-aa14c4271724-konnectivity-ca\") pod \"konnectivity-agent-xfwf2\" (UID: \"791f6cb8-9518-45c6-aad0-aa14c4271724\") " pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:32.711258 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cni-binary-copy\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.711258 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0afbfec3-a252-483c-9b13-9ca1144a1eaf-iptables-alerter-script\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.711258 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-run-netns\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711258 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-systemd\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711258 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-var-lib-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-log-socket\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-env-overrides\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-ovnkube-script-lib\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-kubelet\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-systemd-units\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-node-log\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-cni-netd\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cnibin\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbnc\" (UniqueName: \"kubernetes.io/projected/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-kube-api-access-kpbnc\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-cni-bin\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.711618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-ovnkube-config\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.712286 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-os-release\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.712286 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.712286 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.711686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1dbb731-10bf-478b-bc95-e162549bddd5-ovn-node-metrics-cert\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.713559 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.713536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.714333 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.714238 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 15:40:32.714333 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.714286 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 15:40:32.714661 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.714487 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 15:40:32.714661 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.714564 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gx9k2\"" Jul 07 15:40:32.714661 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.714565 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 15:40:32.715925 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.715906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.716241 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.716135 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 15:40:32.716592 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.716528 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6s466\"" Jul 07 15:40:32.716772 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.716753 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 15:40:32.718610 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.718594 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.718807 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.718784 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 15:40:32.719176 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.719047 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 15:40:32.719336 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.719319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lk648\"" Jul 07 15:40:32.720238 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.720216 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 15:40:32.721219 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.721199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8p26p\"" Jul 07 15:40:32.721463 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.721444 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:40:32.722233 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.721960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.722233 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.722058 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:32.722502 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.722322 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 15:40:32.724464 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.724443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:32.724553 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.724515 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:32.732939 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.732920 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxrsd" Jul 07 15:40:32.742681 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.742660 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxrsd" Jul 07 15:40:32.799301 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.799231 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 15:40:32.805487 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.805439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" event={"ID":"ed097c754df8ae0ea286f9bc978e4472","Type":"ContainerStarted","Data":"e2fbfd2969973a85e1eb1375b505bf7a70bd3aba3b47c8851615fd022e0747bf"} Jul 07 15:40:32.806895 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.806877 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" event={"ID":"8c9f71f5b36a02e94770dabb884fd998","Type":"ContainerStarted","Data":"543f8a393e65b95502462301b9fcc19aabc4c9170b25f01ab39f9918d5729db9"} Jul 07 15:40:32.812248 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-node-log\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.812343 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-k8s-cni-cncf-io\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.812343 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31cd7f20-7bdb-415b-a309-edf730b75623-multus-daemon-config\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.812343 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-modprobe-d\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.812343 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysctl-conf\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.812343 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdrw\" (UniqueName: \"kubernetes.io/projected/dedee6e1-5978-477f-81cf-88e559355fd7-kube-api-access-rpdrw\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-kubelet-config\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cnibin\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbnc\" (UniqueName: \"kubernetes.io/projected/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-kube-api-access-kpbnc\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-cni-bin\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-ovnkube-config\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-os-release\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-os-release\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1dbb731-10bf-478b-bc95-e162549bddd5-ovn-node-metrics-cert\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysconfig\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.812558 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-systemd\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-system-cni-dir\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-etc-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbpw\" (UniqueName: \"kubernetes.io/projected/b1dbb731-10bf-478b-bc95-e162549bddd5-kube-api-access-jkbpw\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-device-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dedee6e1-5978-477f-81cf-88e559355fd7-tmp\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-dbus\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0afbfec3-a252-483c-9b13-9ca1144a1eaf-host-slash\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-multus-certs\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-run\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fx58\" (UniqueName: \"kubernetes.io/projected/0afbfec3-a252-483c-9b13-9ca1144a1eaf-kube-api-access-5fx58\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-ovn\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/791f6cb8-9518-45c6-aad0-aa14c4271724-agent-certs\") pod \"konnectivity-agent-xfwf2\" (UID: \"791f6cb8-9518-45c6-aad0-aa14c4271724\") " pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/791f6cb8-9518-45c6-aad0-aa14c4271724-konnectivity-ca\") pod \"konnectivity-agent-xfwf2\" (UID: \"791f6cb8-9518-45c6-aad0-aa14c4271724\") " pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:32.813037 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31cd7f20-7bdb-415b-a309-edf730b75623-cni-binary-copy\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-registration-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-sys-fs\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cni-binary-copy\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-log-socket\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-ovnkube-script-lib\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-hostroot\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63388bc4-351a-4315-b24f-53c259ab2b4b-hosts-file\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-sys\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.812956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-lib-modules\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-socket-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-host\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-kubelet\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-systemd-units\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-cni-netd\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-kubernetes\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.813593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtjv\" (UniqueName: \"kubernetes.io/projected/500054cf-33d6-4dc7-9dd7-272bbacd6b95-kube-api-access-nmtjv\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-kubelet\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-system-cni-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-cnibin\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dedee6e1-5978-477f-81cf-88e559355fd7-etc-tuned\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-slash\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dsfz\" (UniqueName: \"kubernetes.io/projected/31cd7f20-7bdb-415b-a309-edf730b75623-kube-api-access-7dsfz\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63388bc4-351a-4315-b24f-53c259ab2b4b-tmp-dir\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8hrs\" (UniqueName: \"kubernetes.io/projected/0f7b920c-de3b-4788-81cf-17b3640ed801-kube-api-access-d8hrs\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-netns\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-cni-multus\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-etc-kubernetes\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7b920c-de3b-4788-81cf-17b3640ed801-host\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysctl-d\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-cni-bin\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-conf-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrj5\" (UniqueName: \"kubernetes.io/projected/63388bc4-351a-4315-b24f-53c259ab2b4b-kube-api-access-wcrj5\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0afbfec3-a252-483c-9b13-9ca1144a1eaf-iptables-alerter-script\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-run-netns\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-systemd\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-var-lib-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-env-overrides\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-cni-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-socket-dir-parent\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrxv\" (UniqueName: \"kubernetes.io/projected/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-kube-api-access-7xrxv\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f7b920c-de3b-4788-81cf-17b3640ed801-serviceca\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-var-lib-kubelet\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-node-log\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.813786 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cnibin\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.814001 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-cni-bin\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.814535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.814032 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815014 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.814458 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-ovnkube-config\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815014 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.814542 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-os-release\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.815014 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.814852 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.815135 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815077 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-systemd-units\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815135 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815109 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-var-lib-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815188 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-cni-netd\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815188 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815168 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-etc-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815247 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815191 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-system-cni-dir\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.815247 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815221 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-systemd\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815304 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-openvswitch\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815304 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815273 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815405 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815384 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-env-overrides\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815500 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815167 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 15:40:32.815560 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815518 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-slash\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815705 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0afbfec3-a252-483c-9b13-9ca1144a1eaf-iptables-alerter-script\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.815762 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815735 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-log-socket\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815823 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815769 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-run-netns\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.815823 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0afbfec3-a252-483c-9b13-9ca1144a1eaf-host-slash\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.815919 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815843 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-run-ovn\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.816024 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815963 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.816093 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.815960 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-cni-binary-copy\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.816149 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.816110 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1dbb731-10bf-478b-bc95-e162549bddd5-host-kubelet\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.816240 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.816218 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1dbb731-10bf-478b-bc95-e162549bddd5-ovnkube-script-lib\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.816648 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.816627 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/791f6cb8-9518-45c6-aad0-aa14c4271724-konnectivity-ca\") pod \"konnectivity-agent-xfwf2\" (UID: \"791f6cb8-9518-45c6-aad0-aa14c4271724\") " pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:32.816729 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.816701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.819004 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.818963 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1dbb731-10bf-478b-bc95-e162549bddd5-ovn-node-metrics-cert\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.819098 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.819014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/791f6cb8-9518-45c6-aad0-aa14c4271724-agent-certs\") pod \"konnectivity-agent-xfwf2\" (UID: \"791f6cb8-9518-45c6-aad0-aa14c4271724\") " pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:32.822109 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.822087 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:32.822178 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.822115 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:32.822178 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.822128 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5ftjh for pod openshift-network-diagnostics/network-check-target-ltsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:32.822377 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.822339 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh podName:a93cf57e-bd7f-4a64-9a95-d840b2877541 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:33.322318787 +0000 UTC m=+2.921388060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5ftjh" (UniqueName: "kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh") pod "network-check-target-ltsg9" (UID: "a93cf57e-bd7f-4a64-9a95-d840b2877541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:32.822550 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.822527 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbnc\" (UniqueName: \"kubernetes.io/projected/99d76d9c-ec28-46af-a4a3-cb88c26ec23d-kube-api-access-kpbnc\") pod \"multus-additional-cni-plugins-44mlq\" (UID: \"99d76d9c-ec28-46af-a4a3-cb88c26ec23d\") " pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:32.823638 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.823613 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbpw\" (UniqueName: \"kubernetes.io/projected/b1dbb731-10bf-478b-bc95-e162549bddd5-kube-api-access-jkbpw\") pod \"ovnkube-node-8p85w\" (UID: \"b1dbb731-10bf-478b-bc95-e162549bddd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:32.824743 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.824721 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fx58\" (UniqueName: \"kubernetes.io/projected/0afbfec3-a252-483c-9b13-9ca1144a1eaf-kube-api-access-5fx58\") pod \"iptables-alerter-98ghd\" (UID: \"0afbfec3-a252-483c-9b13-9ca1144a1eaf\") " pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:32.914330 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31cd7f20-7bdb-415b-a309-edf730b75623-cni-binary-copy\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914503 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-registration-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.914503 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-sys-fs\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.914503 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-hostroot\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914503 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63388bc4-351a-4315-b24f-53c259ab2b4b-hosts-file\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.914503 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-sys\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.914503 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-lib-modules\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.914503 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914445 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-registration-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.914503 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-socket-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914522 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-host\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-kubernetes\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914578 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-sys-fs\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtjv\" (UniqueName: \"kubernetes.io/projected/500054cf-33d6-4dc7-9dd7-272bbacd6b95-kube-api-access-nmtjv\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-kubelet\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-system-cni-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-cnibin\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dedee6e1-5978-477f-81cf-88e559355fd7-etc-tuned\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914713 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63388bc4-351a-4315-b24f-53c259ab2b4b-hosts-file\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dsfz\" (UniqueName: \"kubernetes.io/projected/31cd7f20-7bdb-415b-a309-edf730b75623-kube-api-access-7dsfz\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63388bc4-351a-4315-b24f-53c259ab2b4b-tmp-dir\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hrs\" (UniqueName: \"kubernetes.io/projected/0f7b920c-de3b-4788-81cf-17b3640ed801-kube-api-access-d8hrs\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-netns\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-cni-multus\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-etc-kubernetes\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.914933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7b920c-de3b-4788-81cf-17b3640ed801-host\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysctl-d\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31cd7f20-7bdb-415b-a309-edf730b75623-cni-binary-copy\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-cni-bin\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-conf-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914995 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-netns\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915012 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-kubelet\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrj5\" (UniqueName: \"kubernetes.io/projected/63388bc4-351a-4315-b24f-53c259ab2b4b-kube-api-access-wcrj5\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915045 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-sys\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915053 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-socket-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915184 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-lib-modules\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915230 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7b920c-de3b-4788-81cf-17b3640ed801-host\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915235 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-etc-kubernetes\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.914617 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-hostroot\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915282 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-cni-bin\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915299 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63388bc4-351a-4315-b24f-53c259ab2b4b-tmp-dir\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-system-cni-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.915771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915326 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysctl-d\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915327 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-conf-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-host\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-cnibin\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-cni-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-socket-dir-parent\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915531 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-kubernetes\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrxv\" (UniqueName: \"kubernetes.io/projected/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-kube-api-access-7xrxv\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f7b920c-de3b-4788-81cf-17b3640ed801-serviceca\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-var-lib-kubelet\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-k8s-cni-cncf-io\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31cd7f20-7bdb-415b-a309-edf730b75623-multus-daemon-config\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-modprobe-d\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysctl-conf\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdrw\" (UniqueName: \"kubernetes.io/projected/dedee6e1-5978-477f-81cf-88e559355fd7-kube-api-access-rpdrw\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915750 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-cni-dir\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.916599 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-kubelet-config\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-os-release\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-multus-socket-dir-parent\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysconfig\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-systemd\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-device-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dedee6e1-5978-477f-81cf-88e559355fd7-tmp\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-dbus\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-multus-certs\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-run\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f7b920c-de3b-4788-81cf-17b3640ed801-serviceca\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916098 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-modprobe-d\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916105 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-run\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916145 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-kubelet-config\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916165 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysctl-conf\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.916187 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-os-release\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.917369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916210 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-device-dir\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.915720 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-var-lib-kubelet\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.916250 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret podName:27cbb2a7-11c0-43e1-9d8f-1f7e630465d1 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:33.416231161 +0000 UTC m=+3.015300423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret") pod "global-pull-secret-syncer-cvlj7" (UID: "27cbb2a7-11c0-43e1-9d8f-1f7e630465d1") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-var-lib-cni-multus\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-multus-certs\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916275 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31cd7f20-7bdb-415b-a309-edf730b75623-host-run-k8s-cni-cncf-io\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.916289 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-dbus\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916309 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31cd7f20-7bdb-415b-a309-edf730b75623-multus-daemon-config\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:32.916876 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs podName:500054cf-33d6-4dc7-9dd7-272bbacd6b95 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:33.416850794 +0000 UTC m=+3.015920057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs") pod "network-metrics-daemon-27cjl" (UID: "500054cf-33d6-4dc7-9dd7-272bbacd6b95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916314 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-sysconfig\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.918206 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.916335 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dedee6e1-5978-477f-81cf-88e559355fd7-etc-systemd\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.921344 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.921318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dedee6e1-5978-477f-81cf-88e559355fd7-etc-tuned\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.921576 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.921555 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dedee6e1-5978-477f-81cf-88e559355fd7-tmp\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:32.924197 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.924176 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dsfz\" (UniqueName: \"kubernetes.io/projected/31cd7f20-7bdb-415b-a309-edf730b75623-kube-api-access-7dsfz\") pod \"multus-22l6f\" (UID: \"31cd7f20-7bdb-415b-a309-edf730b75623\") " pod="openshift-multus/multus-22l6f" Jul 07 15:40:32.925293 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.925272 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtjv\" (UniqueName: \"kubernetes.io/projected/500054cf-33d6-4dc7-9dd7-272bbacd6b95-kube-api-access-nmtjv\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:32.926386 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.926308 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hrs\" (UniqueName: \"kubernetes.io/projected/0f7b920c-de3b-4788-81cf-17b3640ed801-kube-api-access-d8hrs\") pod \"node-ca-9grq8\" (UID: \"0f7b920c-de3b-4788-81cf-17b3640ed801\") " pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:32.926386 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.926329 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrxv\" (UniqueName: \"kubernetes.io/projected/9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578-kube-api-access-7xrxv\") pod \"aws-ebs-csi-driver-node-vwczw\" (UID: \"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:32.926544 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.926524 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrj5\" (UniqueName: \"kubernetes.io/projected/63388bc4-351a-4315-b24f-53c259ab2b4b-kube-api-access-wcrj5\") pod \"node-resolver-mjdkq\" (UID: \"63388bc4-351a-4315-b24f-53c259ab2b4b\") " pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:32.927652 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:32.927635 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdrw\" (UniqueName: \"kubernetes.io/projected/dedee6e1-5978-477f-81cf-88e559355fd7-kube-api-access-rpdrw\") pod \"tuned-2ll4t\" (UID: \"dedee6e1-5978-477f-81cf-88e559355fd7\") " pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:33.006219 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.006183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-44mlq" Jul 07 15:40:33.015327 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.015302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-98ghd" Jul 07 15:40:33.024071 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.024052 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:33.030623 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.030606 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:40:33.038213 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.038195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22l6f" Jul 07 15:40:33.044713 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.044696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" Jul 07 15:40:33.052228 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.052184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mjdkq" Jul 07 15:40:33.059762 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.059740 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9grq8" Jul 07 15:40:33.064333 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.064316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" Jul 07 15:40:33.418808 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.418729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:33.418808 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.418779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:33.419040 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.418814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:33.419040 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.418927 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:33.419040 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.418942 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:33.419040 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.418942 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:33.419040 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.418954 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:33.419040 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.418987 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5ftjh for pod openshift-network-diagnostics/network-check-target-ltsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:33.419040 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.419016 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs podName:500054cf-33d6-4dc7-9dd7-272bbacd6b95 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:34.41899889 +0000 UTC m=+4.018068153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs") pod "network-metrics-daemon-27cjl" (UID: "500054cf-33d6-4dc7-9dd7-272bbacd6b95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:33.419040 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.419033 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret podName:27cbb2a7-11c0-43e1-9d8f-1f7e630465d1 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:34.419024175 +0000 UTC m=+4.018093437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret") pod "global-pull-secret-syncer-cvlj7" (UID: "27cbb2a7-11c0-43e1-9d8f-1f7e630465d1") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:33.419346 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.419057 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh podName:a93cf57e-bd7f-4a64-9a95-d840b2877541 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:34.419039641 +0000 UTC m=+4.018108912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ftjh" (UniqueName: "kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh") pod "network-check-target-ltsg9" (UID: "a93cf57e-bd7f-4a64-9a95-d840b2877541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:33.746508 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.746407 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:35:32 +0000 UTC" deadline="2028-01-04 22:46:00.014476266 +0000 UTC" Jul 07 15:40:33.746508 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.746467 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13111h5m26.268014101s" Jul 07 15:40:33.800880 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:33.800853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:33.801061 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:33.800981 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:33.880791 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:33.880767 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d477c4c_0b26_4aa3_ba2a_adcb6c9aa578.slice/crio-138d52eb733253dbc3336b6c308895fa5ab703443fd19857464a639acc5c5d4f WatchSource:0}: Error finding container 138d52eb733253dbc3336b6c308895fa5ab703443fd19857464a639acc5c5d4f: Status 404 returned error can't find the container with id 138d52eb733253dbc3336b6c308895fa5ab703443fd19857464a639acc5c5d4f Jul 07 15:40:33.882148 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:33.882126 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1dbb731_10bf_478b_bc95_e162549bddd5.slice/crio-e0f43b4e3f1d82de39d06cd1fbb72f9a706e5a8eb9d3e7825821f6a990eb1995 WatchSource:0}: Error finding container e0f43b4e3f1d82de39d06cd1fbb72f9a706e5a8eb9d3e7825821f6a990eb1995: Status 404 returned error can't find the container with id e0f43b4e3f1d82de39d06cd1fbb72f9a706e5a8eb9d3e7825821f6a990eb1995 Jul 07 15:40:33.884582 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:33.884520 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afbfec3_a252_483c_9b13_9ca1144a1eaf.slice/crio-bf5f1427997e4866cc2e4ff046fcee73509868a023ff5a327d5de92b3ae409a0 WatchSource:0}: Error finding container bf5f1427997e4866cc2e4ff046fcee73509868a023ff5a327d5de92b3ae409a0: Status 404 returned error can't find the container with id bf5f1427997e4866cc2e4ff046fcee73509868a023ff5a327d5de92b3ae409a0 Jul 07 15:40:33.912337 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:33.912301 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d76d9c_ec28_46af_a4a3_cb88c26ec23d.slice/crio-d5b1c4c9e79e3435e052aac7de558a85a8321be5ed9f5b24b841eeca47cf5b13 WatchSource:0}: Error finding container d5b1c4c9e79e3435e052aac7de558a85a8321be5ed9f5b24b841eeca47cf5b13: Status 404 returned error can't find the container with id d5b1c4c9e79e3435e052aac7de558a85a8321be5ed9f5b24b841eeca47cf5b13 Jul 07 15:40:33.913143 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:33.913119 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791f6cb8_9518_45c6_aad0_aa14c4271724.slice/crio-0a746c0d63c026126efb21c34b77e8f4adc06debcd2ca245baa19b79f766cd5f WatchSource:0}: Error finding container 0a746c0d63c026126efb21c34b77e8f4adc06debcd2ca245baa19b79f766cd5f: Status 404 returned error can't find the container with id 0a746c0d63c026126efb21c34b77e8f4adc06debcd2ca245baa19b79f766cd5f Jul 07 15:40:33.914019 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:33.913996 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedee6e1_5978_477f_81cf_88e559355fd7.slice/crio-2720e63d250daf93e99ad2c7bb3303dcaa05dab2dc0ec4d907d665bacf56054d WatchSource:0}: Error finding container 2720e63d250daf93e99ad2c7bb3303dcaa05dab2dc0ec4d907d665bacf56054d: Status 404 returned error can't find the container with id 2720e63d250daf93e99ad2c7bb3303dcaa05dab2dc0ec4d907d665bacf56054d Jul 07 15:40:33.914897 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:40:33.914878 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cd7f20_7bdb_415b_a309_edf730b75623.slice/crio-c2f5911e599c3142191bc58bcb323a4e86921323906c266f1a8800914848edeb WatchSource:0}: Error finding container c2f5911e599c3142191bc58bcb323a4e86921323906c266f1a8800914848edeb: Status 404 returned error can't find the container with id c2f5911e599c3142191bc58bcb323a4e86921323906c266f1a8800914848edeb Jul 07 15:40:34.426812 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.426518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:34.426812 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.426787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.426821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.426714 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.426879 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.426899 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5ftjh for pod openshift-network-diagnostics/network-check-target-ltsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.426947 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.426948 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.426985 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh podName:a93cf57e-bd7f-4a64-9a95-d840b2877541 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:36.426946301 +0000 UTC m=+6.026015586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ftjh" (UniqueName: "kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh") pod "network-check-target-ltsg9" (UID: "a93cf57e-bd7f-4a64-9a95-d840b2877541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.427017 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs podName:500054cf-33d6-4dc7-9dd7-272bbacd6b95 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:36.427001985 +0000 UTC m=+6.026071264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs") pod "network-metrics-daemon-27cjl" (UID: "500054cf-33d6-4dc7-9dd7-272bbacd6b95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:34.427054 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.427042 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret podName:27cbb2a7-11c0-43e1-9d8f-1f7e630465d1 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:36.427032757 +0000 UTC m=+6.026102029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret") pod "global-pull-secret-syncer-cvlj7" (UID: "27cbb2a7-11c0-43e1-9d8f-1f7e630465d1") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:34.746907 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.746774 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:35:32 +0000 UTC" deadline="2028-01-25 10:25:18.683036229 +0000 UTC" Jul 07 15:40:34.746907 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.746828 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13602h44m43.93621254s" Jul 07 15:40:34.803885 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.803246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:34.803885 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.803365 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:34.803885 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.803449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:34.803885 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:34.803530 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:34.823202 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.823169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mjdkq" event={"ID":"63388bc4-351a-4315-b24f-53c259ab2b4b","Type":"ContainerStarted","Data":"90ac4ad1107b5285bd6d4ee4b36ef98e5f5d12f7e36b41e784aa568fd1ab5aa1"} Jul 07 15:40:34.837320 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.837286 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"e0f43b4e3f1d82de39d06cd1fbb72f9a706e5a8eb9d3e7825821f6a990eb1995"} Jul 07 15:40:34.844865 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.844828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9grq8" event={"ID":"0f7b920c-de3b-4788-81cf-17b3640ed801","Type":"ContainerStarted","Data":"4590c866c12379ed2b37d5db4035afda1f2cd1c081908a44ed19e1dc16a061cc"} Jul 07 15:40:34.854840 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.854809 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" event={"ID":"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578","Type":"ContainerStarted","Data":"138d52eb733253dbc3336b6c308895fa5ab703443fd19857464a639acc5c5d4f"} Jul 07 15:40:34.859769 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.859693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22l6f" event={"ID":"31cd7f20-7bdb-415b-a309-edf730b75623","Type":"ContainerStarted","Data":"c2f5911e599c3142191bc58bcb323a4e86921323906c266f1a8800914848edeb"} Jul 07 15:40:34.866302 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.866256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xfwf2" event={"ID":"791f6cb8-9518-45c6-aad0-aa14c4271724","Type":"ContainerStarted","Data":"0a746c0d63c026126efb21c34b77e8f4adc06debcd2ca245baa19b79f766cd5f"} Jul 07 15:40:34.869789 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.867937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44mlq" event={"ID":"99d76d9c-ec28-46af-a4a3-cb88c26ec23d","Type":"ContainerStarted","Data":"d5b1c4c9e79e3435e052aac7de558a85a8321be5ed9f5b24b841eeca47cf5b13"} Jul 07 15:40:34.877354 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.877326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-98ghd" event={"ID":"0afbfec3-a252-483c-9b13-9ca1144a1eaf","Type":"ContainerStarted","Data":"bf5f1427997e4866cc2e4ff046fcee73509868a023ff5a327d5de92b3ae409a0"} Jul 07 15:40:34.881539 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.881499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" event={"ID":"ed097c754df8ae0ea286f9bc978e4472","Type":"ContainerStarted","Data":"1ccc1913523060fbd8366de5944db5267ebdb4f12ff864e7601d49e85575de93"} Jul 07 15:40:34.894499 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.894472 2576 generic.go:356] "Generic (PLEG): container finished" podID="8c9f71f5b36a02e94770dabb884fd998" containerID="74366cdf945d5f7ddcf4373cb44047a79924439db4223302a99c6f5104746f8c" exitCode=0 Jul 07 15:40:34.894601 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.894549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" event={"ID":"8c9f71f5b36a02e94770dabb884fd998","Type":"ContainerDied","Data":"74366cdf945d5f7ddcf4373cb44047a79924439db4223302a99c6f5104746f8c"} Jul 07 15:40:34.907205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.907180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" event={"ID":"dedee6e1-5978-477f-81cf-88e559355fd7","Type":"ContainerStarted","Data":"2720e63d250daf93e99ad2c7bb3303dcaa05dab2dc0ec4d907d665bacf56054d"} Jul 07 15:40:34.914589 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:34.914541 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.ec2.internal" podStartSLOduration=2.914524437 podStartE2EDuration="2.914524437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:34.898950334 +0000 UTC m=+4.498019599" watchObservedRunningTime="2026-07-07 15:40:34.914524437 +0000 UTC m=+4.513593722" Jul 07 15:40:35.801519 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:35.801488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:35.802007 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:35.801617 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:35.923956 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:35.923387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" event={"ID":"8c9f71f5b36a02e94770dabb884fd998","Type":"ContainerStarted","Data":"dffc3630d52cb3c21d14479fc7673894a196a3c74e22d93a0ebb9ab6a4add8c5"} Jul 07 15:40:36.444658 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:36.444502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:36.444658 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:36.444551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:36.444658 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:36.444574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:36.444882 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.444677 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:36.444882 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.444680 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:36.444882 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.444691 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:36.444882 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.444704 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:36.444882 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.444717 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5ftjh for pod openshift-network-diagnostics/network-check-target-ltsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:36.444882 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.444725 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs podName:500054cf-33d6-4dc7-9dd7-272bbacd6b95 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:40.444711649 +0000 UTC m=+10.043780912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs") pod "network-metrics-daemon-27cjl" (UID: "500054cf-33d6-4dc7-9dd7-272bbacd6b95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:36.444882 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.444750 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret podName:27cbb2a7-11c0-43e1-9d8f-1f7e630465d1 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:40.444731957 +0000 UTC m=+10.043801225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret") pod "global-pull-secret-syncer-cvlj7" (UID: "27cbb2a7-11c0-43e1-9d8f-1f7e630465d1") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:36.444882 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.444771 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh podName:a93cf57e-bd7f-4a64-9a95-d840b2877541 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:40.444760466 +0000 UTC m=+10.043829730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ftjh" (UniqueName: "kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh") pod "network-check-target-ltsg9" (UID: "a93cf57e-bd7f-4a64-9a95-d840b2877541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:36.803161 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:36.803074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:36.803660 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.803267 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:36.803754 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:36.803729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:36.803865 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:36.803840 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:37.800955 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:37.800921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:37.801136 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:37.801074 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:38.803710 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:38.803074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:38.803710 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:38.803197 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:38.803710 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:38.803525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:38.803710 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:38.803643 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:39.801547 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:39.801515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:39.801717 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:39.801648 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:40.477458 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:40.477416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:40.477915 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:40.477470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:40.477915 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:40.477506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:40.477915 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.477682 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:40.477915 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.477747 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs podName:500054cf-33d6-4dc7-9dd7-272bbacd6b95 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:48.477727875 +0000 UTC m=+18.076797141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs") pod "network-metrics-daemon-27cjl" (UID: "500054cf-33d6-4dc7-9dd7-272bbacd6b95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:40.478505 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.478267 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:40.478505 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.478293 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:40.478505 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.478307 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5ftjh for pod openshift-network-diagnostics/network-check-target-ltsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:40.478505 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.478350 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh podName:a93cf57e-bd7f-4a64-9a95-d840b2877541 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:48.478336076 +0000 UTC m=+18.077405340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ftjh" (UniqueName: "kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh") pod "network-check-target-ltsg9" (UID: "a93cf57e-bd7f-4a64-9a95-d840b2877541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:40.478505 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.478412 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:40.478505 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.478447 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret podName:27cbb2a7-11c0-43e1-9d8f-1f7e630465d1 nodeName:}" failed. No retries permitted until 2026-07-07 15:40:48.478433019 +0000 UTC m=+18.077502284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret") pod "global-pull-secret-syncer-cvlj7" (UID: "27cbb2a7-11c0-43e1-9d8f-1f7e630465d1") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:40.801483 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:40.801393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:40.801679 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.801512 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:40.802043 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:40.802025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:40.802143 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:40.802124 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:41.801366 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:41.801324 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:41.801833 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:41.801460 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:42.803172 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:42.803138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:42.803653 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:42.803267 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:42.803653 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:42.803633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:42.803765 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:42.803733 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:43.801074 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:43.801041 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:43.801241 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:43.801160 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:44.801061 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:44.801026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:44.801524 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:44.801228 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:44.801627 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:44.801030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:44.801763 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:44.801741 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:45.801270 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:45.801234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:45.801744 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:45.801361 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:46.801134 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:46.801093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:46.801320 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:46.801092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:46.801320 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:46.801295 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:46.801716 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:46.801409 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:47.801205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:47.801168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:47.801431 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:47.801301 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:48.537805 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:48.537767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:48.537962 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:48.537815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:48.537962 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:48.537847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:48.537962 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.537913 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:48.537962 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.537926 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:40:48.537962 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.537945 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:40:48.537962 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.537954 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:48.537962 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.537958 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5ftjh for pod openshift-network-diagnostics/network-check-target-ltsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:48.538214 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.538009 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret podName:27cbb2a7-11c0-43e1-9d8f-1f7e630465d1 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:04.537989884 +0000 UTC m=+34.137059149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret") pod "global-pull-secret-syncer-cvlj7" (UID: "27cbb2a7-11c0-43e1-9d8f-1f7e630465d1") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:40:48.538214 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.538030 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh podName:a93cf57e-bd7f-4a64-9a95-d840b2877541 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:04.538015631 +0000 UTC m=+34.137084894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ftjh" (UniqueName: "kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh") pod "network-check-target-ltsg9" (UID: "a93cf57e-bd7f-4a64-9a95-d840b2877541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:40:48.538214 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.538044 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs podName:500054cf-33d6-4dc7-9dd7-272bbacd6b95 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:04.538038125 +0000 UTC m=+34.137107387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs") pod "network-metrics-daemon-27cjl" (UID: "500054cf-33d6-4dc7-9dd7-272bbacd6b95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:40:48.800948 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:48.800863 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:48.801126 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.801004 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:48.801126 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:48.801069 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:48.801233 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:48.801191 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:49.801207 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:49.801176 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:49.801632 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:49.801306 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:50.802229 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:50.802191 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:50.802627 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:50.802321 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:50.802627 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:50.802387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:50.802627 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:50.802498 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:51.801518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:51.801479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:51.801702 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:51.801614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:52.801697 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:52.801656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:52.802301 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:52.801792 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:52.802301 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:52.801837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:52.802301 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:52.802021 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:53.801893 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.801696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:53.802756 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:53.802037 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:53.963254 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.963178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22l6f" event={"ID":"31cd7f20-7bdb-415b-a309-edf730b75623","Type":"ContainerStarted","Data":"c6de4313c0fc94678221604580ae3a6812fc4454c812355c00f38e7f18ab8ade"} Jul 07 15:40:53.964942 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.964898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xfwf2" event={"ID":"791f6cb8-9518-45c6-aad0-aa14c4271724","Type":"ContainerStarted","Data":"611f99f6af70938cb5fe9faef4298acc1be99ad1db31946eba913e70e49c212e"} Jul 07 15:40:53.966775 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.966660 2576 generic.go:356] "Generic (PLEG): container finished" podID="99d76d9c-ec28-46af-a4a3-cb88c26ec23d" containerID="fa9f7cdaac41405639c487f1db785fdd213d4b035d095d8526914a896d71961e" exitCode=0 Jul 07 15:40:53.966775 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.966745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44mlq" event={"ID":"99d76d9c-ec28-46af-a4a3-cb88c26ec23d","Type":"ContainerDied","Data":"fa9f7cdaac41405639c487f1db785fdd213d4b035d095d8526914a896d71961e"} Jul 07 15:40:53.968395 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.968310 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" event={"ID":"dedee6e1-5978-477f-81cf-88e559355fd7","Type":"ContainerStarted","Data":"bd54ebfa87992b79dbf3ed5ea14276c138cb5f801d8a9159080ae14042d63e44"} Jul 07 15:40:53.970023 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.970000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mjdkq" event={"ID":"63388bc4-351a-4315-b24f-53c259ab2b4b","Type":"ContainerStarted","Data":"fe298ef4bc36e49fc8065dcd13eca9779dca9156a985becaedf8003a4a086c8c"} Jul 07 15:40:53.973272 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.973251 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"56e7ed6baaa16d79930706512a238f3530b5bd9705e9b8349db6d94237a90c7c"} Jul 07 15:40:53.973384 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.973317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"a64c188d5d5fdfc03de3075a95a26ade7dc0338ab4775d1a9814d24c6b079ae8"} Jul 07 15:40:53.973384 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.973361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"bbd8758e8e033b2f45a823b6a196cad400cee25b2b08eb5830901ccfb8660eaa"} Jul 07 15:40:53.973491 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.973408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"e10af344ef9c5b49c526ca3a5cfb3bdae126df0fd9a098fc19e86e445a9c5187"} Jul 07 15:40:53.973491 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.973448 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"6a300885cca0de57cbe12d8cebca3dc6841c3df286d60baefb34ef6b5335842b"} Jul 07 15:40:53.973491 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.973486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"778a8b690739354e1ba6b5cfcd8070444b1d822aab4c1f3367d749a47ca4eb53"} Jul 07 15:40:53.975220 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.974823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9grq8" event={"ID":"0f7b920c-de3b-4788-81cf-17b3640ed801","Type":"ContainerStarted","Data":"43792b72a22540a4786a626a82f21ad6df6c6f2623ff763fa7fd0ba79e9ece84"} Jul 07 15:40:53.976490 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.976469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" event={"ID":"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578","Type":"ContainerStarted","Data":"f21aef37d21b8fa947fc3727b9d7a56047ac294897c594c5b36b551b90b63fe5"} Jul 07 15:40:53.983373 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:53.983333 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.ec2.internal" podStartSLOduration=21.983321171 podStartE2EDuration="21.983321171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:35.942221827 +0000 UTC m=+5.541291136" watchObservedRunningTime="2026-07-07 15:40:53.983321171 +0000 UTC m=+23.582390455" Jul 07 15:40:54.000956 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.000915 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-22l6f" podStartSLOduration=4.97241667 podStartE2EDuration="24.000900513s" totalImagesPullingTime="19.028483843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:53.983783945 +0000 UTC m=+23.582853228" watchObservedRunningTime="2026-07-07 15:40:54.000900513 +0000 UTC m=+23.599969801" Jul 07 15:40:54.025576 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.025525 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xfwf2" podStartSLOduration=13.21770281 podStartE2EDuration="24.025504097s" totalImagesPullingTime="10.807801287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:54.008868059 +0000 UTC m=+23.607937353" watchObservedRunningTime="2026-07-07 15:40:54.025504097 +0000 UTC m=+23.624573385" Jul 07 15:40:54.046878 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.046828 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2ll4t" podStartSLOduration=5.068001304 podStartE2EDuration="24.046813578s" totalImagesPullingTime="18.978812274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:54.046714814 +0000 UTC m=+23.645784117" watchObservedRunningTime="2026-07-07 15:40:54.046813578 +0000 UTC m=+23.645882862" Jul 07 15:40:54.047415 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.047385 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mjdkq" podStartSLOduration=5.063648961 podStartE2EDuration="24.047377045s" totalImagesPullingTime="18.983728084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:54.025117045 +0000 UTC m=+23.624186330" watchObservedRunningTime="2026-07-07 15:40:54.047377045 +0000 UTC m=+23.646446385" Jul 07 15:40:54.096755 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.096703 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9grq8" podStartSLOduration=5.112605934 podStartE2EDuration="24.09668967s" totalImagesPullingTime="18.984083736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:54.096219305 +0000 UTC m=+23.695288589" watchObservedRunningTime="2026-07-07 15:40:54.09668967 +0000 UTC m=+23.695758954" Jul 07 15:40:54.466766 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.466716 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 15:40:54.776337 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.776187 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T15:40:54.466736204Z","UUID":"432d0b9c-5756-4243-9634-18982bb198e7","Handler":null,"Name":"","Endpoint":""} Jul 07 15:40:54.777758 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.777733 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 15:40:54.777877 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.777782 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 15:40:54.801655 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.801627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:54.801820 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.801640 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:54.801820 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:54.801749 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:54.801943 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:54.801831 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:54.981428 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.981389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" event={"ID":"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578","Type":"ContainerStarted","Data":"ef6dbe36b515d6b4ca944e3f17e7ccd657c9072e7febb3a197f35209ba8917c6"} Jul 07 15:40:54.985489 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:54.984016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-98ghd" event={"ID":"0afbfec3-a252-483c-9b13-9ca1144a1eaf","Type":"ContainerStarted","Data":"f51c2e0c64d6c3647cef3e918330bd2bef07ff1d1b8c26bace8a100692fc31e4"} Jul 07 15:40:55.800630 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:55.800605 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:55.800797 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:55.800731 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:55.989867 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:55.989576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"86cdea09948a1f5817327b79fc63a46cd4a642612704a3a64900f15d22abdab2"} Jul 07 15:40:55.991846 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:55.991821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" event={"ID":"9d477c4c-0b26-4aa3-ba2a-adcb6c9aa578","Type":"ContainerStarted","Data":"43b47b82d2a870edb97880e0331cf33a78436cd8e7c0a85004174ec8dd738513"} Jul 07 15:40:56.013495 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:56.013446 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwczw" podStartSLOduration=4.573304574 podStartE2EDuration="26.01343028s" totalImagesPullingTime="21.440125706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:56.011828169 +0000 UTC m=+25.610897454" watchObservedRunningTime="2026-07-07 15:40:56.01343028 +0000 UTC m=+25.612499563" Jul 07 15:40:56.014142 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:56.014101 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-98ghd" podStartSLOduration=7.00655875 podStartE2EDuration="26.014088328s" totalImagesPullingTime="19.007529578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:55.00205348 +0000 UTC m=+24.601122789" watchObservedRunningTime="2026-07-07 15:40:56.014088328 +0000 UTC m=+25.613157608" Jul 07 15:40:56.483202 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:56.483157 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:56.483742 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:56.483724 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:56.804737 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:56.804668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:56.804928 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:56.804771 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:56.804928 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:56.804834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:56.805063 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:56.804941 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:56.994797 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:56.994751 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:56.995244 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:56.995055 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xfwf2" Jul 07 15:40:57.801415 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:57.801386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:57.801609 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:57.801502 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:58.002370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:58.002126 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" event={"ID":"b1dbb731-10bf-478b-bc95-e162549bddd5","Type":"ContainerStarted","Data":"3f10d6db7b685a26d1cf20690b315de8de574ac5dee6bfa5ba8e2cf7d54ea8c9"} Jul 07 15:40:58.804394 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:58.804204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:58.804568 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:58.804253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:58.804568 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:58.804491 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:58.804682 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:58.804588 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:59.006104 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.006021 2576 generic.go:356] "Generic (PLEG): container finished" podID="99d76d9c-ec28-46af-a4a3-cb88c26ec23d" containerID="a2b351dc295f480698a224397d180f31b79d7d6f4aa46630a299caf33b92a790" exitCode=0 Jul 07 15:40:59.006509 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.006103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44mlq" event={"ID":"99d76d9c-ec28-46af-a4a3-cb88c26ec23d","Type":"ContainerDied","Data":"a2b351dc295f480698a224397d180f31b79d7d6f4aa46630a299caf33b92a790"} Jul 07 15:40:59.060791 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.060730 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" podStartSLOduration=9.992541094 podStartE2EDuration="29.060717002s" totalImagesPullingTime="19.068175908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:40:59.060280576 +0000 UTC m=+28.659349859" watchObservedRunningTime="2026-07-07 15:40:59.060717002 +0000 UTC m=+28.659786286" Jul 07 15:40:59.801104 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.801071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:40:59.801287 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:59.801175 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:40:59.922123 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.922087 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ltsg9"] Jul 07 15:40:59.922361 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.922219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:40:59.922361 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:59.922330 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:40:59.925086 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.925055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-27cjl"] Jul 07 15:40:59.925265 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.925249 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:40:59.925380 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:40:59.925359 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:40:59.925700 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:40:59.925679 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cvlj7"] Jul 07 15:41:00.008786 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:00.008756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:41:00.009338 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:00.008926 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:41:00.009399 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:00.009362 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:41:00.009450 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:00.009435 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:41:00.009496 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:00.009476 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:41:00.028796 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:00.028771 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:41:00.028934 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:00.028872 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:41:01.012941 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:01.012734 2576 generic.go:356] "Generic (PLEG): container finished" podID="99d76d9c-ec28-46af-a4a3-cb88c26ec23d" containerID="3e14cdfd411b89fe309b75ef7dbe23e9e9c07a563f9d342ce9ca9c339ff301b0" exitCode=0 Jul 07 15:41:01.013318 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:01.012825 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44mlq" event={"ID":"99d76d9c-ec28-46af-a4a3-cb88c26ec23d","Type":"ContainerDied","Data":"3e14cdfd411b89fe309b75ef7dbe23e9e9c07a563f9d342ce9ca9c339ff301b0"} Jul 07 15:41:01.800884 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:01.800853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:41:01.800884 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:01.800893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:41:01.801116 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:01.800853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:41:01.801116 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:01.801003 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:41:01.801116 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:01.801071 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:41:01.801239 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:01.801173 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:41:02.028416 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:02.028369 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" podUID="b1dbb731-10bf-478b-bc95-e162549bddd5" containerName="ovnkube-controller" probeResult="failure" output="" Jul 07 15:41:03.020535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:03.020445 2576 generic.go:356] "Generic (PLEG): container finished" podID="99d76d9c-ec28-46af-a4a3-cb88c26ec23d" containerID="870dc653ebe86698ffb1a4b1deb744bd92268573a3769c5aacd0e892773ae13a" exitCode=0 Jul 07 15:41:03.020535 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:03.020501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44mlq" event={"ID":"99d76d9c-ec28-46af-a4a3-cb88c26ec23d","Type":"ContainerDied","Data":"870dc653ebe86698ffb1a4b1deb744bd92268573a3769c5aacd0e892773ae13a"} Jul 07 15:41:03.801457 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:03.801423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:41:03.801457 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:03.801445 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:41:03.802034 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:03.801467 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:41:03.802034 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:03.801549 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltsg9" podUID="a93cf57e-bd7f-4a64-9a95-d840b2877541" Jul 07 15:41:03.802034 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:03.801617 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvlj7" podUID="27cbb2a7-11c0-43e1-9d8f-1f7e630465d1" Jul 07 15:41:03.802034 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:03.801690 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27cjl" podUID="500054cf-33d6-4dc7-9dd7-272bbacd6b95" Jul 07 15:41:04.566616 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:04.566574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:41:04.566807 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:04.566626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:41:04.566807 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:04.566732 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:04.566807 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:04.566761 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:04.566807 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:04.566786 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:04.566807 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:04.566793 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret podName:27cbb2a7-11c0-43e1-9d8f-1f7e630465d1 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:36.566775328 +0000 UTC m=+66.165844593 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret") pod "global-pull-secret-syncer-cvlj7" (UID: "27cbb2a7-11c0-43e1-9d8f-1f7e630465d1") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:04.566807 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:04.566797 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5ftjh for pod openshift-network-diagnostics/network-check-target-ltsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:04.567152 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:04.566843 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh podName:a93cf57e-bd7f-4a64-9a95-d840b2877541 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:36.566825797 +0000 UTC m=+66.165895060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ftjh" (UniqueName: "kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh") pod "network-check-target-ltsg9" (UID: "a93cf57e-bd7f-4a64-9a95-d840b2877541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:04.567152 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:04.566864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:41:04.567152 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:04.566954 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:04.567152 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:04.567018 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs podName:500054cf-33d6-4dc7-9dd7-272bbacd6b95 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:36.567005059 +0000 UTC m=+66.166074320 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs") pod "network-metrics-daemon-27cjl" (UID: "500054cf-33d6-4dc7-9dd7-272bbacd6b95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:04.909287 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:04.909210 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mjdkq_63388bc4-351a-4315-b24f-53c259ab2b4b/dns-node-resolver/0.log" Jul 07 15:41:05.483856 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.483827 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9grq8_0f7b920c-de3b-4788-81cf-17b3640ed801/node-ca/0.log" Jul 07 15:41:05.750920 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.750837 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-175.ec2.internal" event="NodeReady" Jul 07 15:41:05.751096 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.751008 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 15:41:05.791888 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.791859 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dcc5bb45d-l9tjf"] Jul 07 15:41:05.797026 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.797000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.799860 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.799827 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 15:41:05.800031 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.799867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 15:41:05.800181 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.800161 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 15:41:05.800324 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.800306 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdgps\"" Jul 07 15:41:05.800602 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.800585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:41:05.800728 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.800699 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:41:05.800837 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.800734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:41:05.804473 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.804399 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 15:41:05.804681 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.804631 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 15:41:05.804681 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.804642 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 15:41:05.804816 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.804635 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 15:41:05.804816 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.804790 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v8g69\"" Jul 07 15:41:05.804891 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.804857 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jh4tr\"" Jul 07 15:41:05.810465 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.810236 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dcc5bb45d-l9tjf"] Jul 07 15:41:05.811692 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.811661 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 15:41:05.812998 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.812365 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w6j8h"] Jul 07 15:41:05.816680 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.816657 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4b46w"] Jul 07 15:41:05.816805 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.816786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w6j8h" Jul 07 15:41:05.819536 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.819515 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 15:41:05.819672 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.819652 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tqcfj\"" Jul 07 15:41:05.819783 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.819687 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 15:41:05.819888 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.819870 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 15:41:05.820756 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.820739 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:05.825330 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.825307 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 15:41:05.825438 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.825344 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 15:41:05.825438 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.825344 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fwwjf\"" Jul 07 15:41:05.825618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.825602 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 15:41:05.825618 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.825616 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 15:41:05.836019 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.836000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w6j8h"] Jul 07 15:41:05.836138 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.836041 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4b46w"] Jul 07 15:41:05.836814 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.836790 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tg7zm"] Jul 07 15:41:05.840853 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.840836 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:05.844296 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.844276 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 15:41:05.844423 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.844276 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 15:41:05.844651 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.844634 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tn46g\"" Jul 07 15:41:05.854226 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.854205 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tg7zm"] Jul 07 15:41:05.877224 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.877196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-certificates\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.877362 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.877227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-installation-pull-secrets\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.877362 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.877260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-image-registry-private-configuration\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.877362 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.877287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.877499 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.877356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-trusted-ca\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.877499 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.877466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2970c78e-3018-4a3e-8f54-73e26dca206a-ca-trust-extracted\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.877499 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.877492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rbd\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-kube-api-access-q6rbd\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.877638 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.877538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-bound-sa-token\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.978513 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2970c78e-3018-4a3e-8f54-73e26dca206a-ca-trust-extracted\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.978513 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rbd\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-kube-api-access-q6rbd\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxqh\" (UniqueName: \"kubernetes.io/projected/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-api-access-qlxqh\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-bound-sa-token\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-certificates\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5155845-d5c0-44b1-a045-9ae36b9bbfb9-cert\") pod \"ingress-canary-w6j8h\" (UID: \"f5155845-d5c0-44b1-a045-9ae36b9bbfb9\") " pod="openshift-ingress-canary/ingress-canary-w6j8h" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978849 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2970c78e-3018-4a3e-8f54-73e26dca206a-ca-trust-extracted\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.978875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hghx\" (UniqueName: \"kubernetes.io/projected/f5155845-d5c0-44b1-a045-9ae36b9bbfb9-kube-api-access-7hghx\") pod \"ingress-canary-w6j8h\" (UID: \"f5155845-d5c0-44b1-a045-9ae36b9bbfb9\") " pod="openshift-ingress-canary/ingress-canary-w6j8h" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-installation-pull-secrets\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:05.979060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-image-registry-private-configuration\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d079eb86-acc0-4b1d-9cb5-873b28271068-tmp-dir\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgvw\" (UniqueName: \"kubernetes.io/projected/d079eb86-acc0-4b1d-9cb5-873b28271068-kube-api-access-cqgvw\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc0f2445-5253-4dec-abeb-f60ea932d598-crio-socket\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc0f2445-5253-4dec-abeb-f60ea932d598-data-volume\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-trusted-ca\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d079eb86-acc0-4b1d-9cb5-873b28271068-config-volume\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.979415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-certificates\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:05.979518 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:41:05.979552 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:05.979538 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcc5bb45d-l9tjf: secret "image-registry-tls" not found Jul 07 15:41:05.980006 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:05.979669 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls podName:2970c78e-3018-4a3e-8f54-73e26dca206a nodeName:}" failed. No retries permitted until 2026-07-07 15:41:06.479648447 +0000 UTC m=+36.078717723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls") pod "image-registry-5dcc5bb45d-l9tjf" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a") : secret "image-registry-tls" not found Jul 07 15:41:05.980359 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.980338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-trusted-ca\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.983439 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.983418 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-installation-pull-secrets\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.983555 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.983425 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-image-registry-private-configuration\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.988043 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.987995 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rbd\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-kube-api-access-q6rbd\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:05.989344 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:05.989321 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-bound-sa-token\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:06.080288 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5155845-d5c0-44b1-a045-9ae36b9bbfb9-cert\") pod \"ingress-canary-w6j8h\" (UID: \"f5155845-d5c0-44b1-a045-9ae36b9bbfb9\") " pod="openshift-ingress-canary/ingress-canary-w6j8h" Jul 07 15:41:06.080474 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hghx\" (UniqueName: \"kubernetes.io/projected/f5155845-d5c0-44b1-a045-9ae36b9bbfb9-kube-api-access-7hghx\") pod \"ingress-canary-w6j8h\" (UID: \"f5155845-d5c0-44b1-a045-9ae36b9bbfb9\") " pod="openshift-ingress-canary/ingress-canary-w6j8h" Jul 07 15:41:06.080474 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.080474 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d079eb86-acc0-4b1d-9cb5-873b28271068-tmp-dir\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:06.080474 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.080459 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:06.080666 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.080536 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm podName:dc0f2445-5253-4dec-abeb-f60ea932d598 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:06.580513783 +0000 UTC m=+36.179583078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4b46w" (UID: "dc0f2445-5253-4dec-abeb-f60ea932d598") : configmap "kube-rbac-proxy" not found Jul 07 15:41:06.080666 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgvw\" (UniqueName: \"kubernetes.io/projected/d079eb86-acc0-4b1d-9cb5-873b28271068-kube-api-access-cqgvw\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:06.080666 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d079eb86-acc0-4b1d-9cb5-873b28271068-tmp-dir\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:06.080666 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc0f2445-5253-4dec-abeb-f60ea932d598-crio-socket\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.080855 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc0f2445-5253-4dec-abeb-f60ea932d598-data-volume\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.080855 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:06.080855 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d079eb86-acc0-4b1d-9cb5-873b28271068-config-volume\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:06.080855 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.081087 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.080861 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:06.081087 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxqh\" (UniqueName: \"kubernetes.io/projected/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-api-access-qlxqh\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.081087 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.080932 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls podName:d079eb86-acc0-4b1d-9cb5-873b28271068 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:06.580912529 +0000 UTC m=+36.179981806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls") pod "dns-default-tg7zm" (UID: "d079eb86-acc0-4b1d-9cb5-873b28271068") : secret "dns-default-metrics-tls" not found Jul 07 15:41:06.081087 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc0f2445-5253-4dec-abeb-f60ea932d598-crio-socket\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.081087 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.080952 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc0f2445-5253-4dec-abeb-f60ea932d598-data-volume\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.081087 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.081014 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:06.081087 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.081090 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls podName:dc0f2445-5253-4dec-abeb-f60ea932d598 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:06.581073082 +0000 UTC m=+36.180142358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b46w" (UID: "dc0f2445-5253-4dec-abeb-f60ea932d598") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:06.081485 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.081465 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d079eb86-acc0-4b1d-9cb5-873b28271068-config-volume\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:06.083435 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.083410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5155845-d5c0-44b1-a045-9ae36b9bbfb9-cert\") pod \"ingress-canary-w6j8h\" (UID: \"f5155845-d5c0-44b1-a045-9ae36b9bbfb9\") " pod="openshift-ingress-canary/ingress-canary-w6j8h" Jul 07 15:41:06.093127 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.093106 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgvw\" (UniqueName: \"kubernetes.io/projected/d079eb86-acc0-4b1d-9cb5-873b28271068-kube-api-access-cqgvw\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:06.094959 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.094941 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hghx\" (UniqueName: \"kubernetes.io/projected/f5155845-d5c0-44b1-a045-9ae36b9bbfb9-kube-api-access-7hghx\") pod \"ingress-canary-w6j8h\" (UID: \"f5155845-d5c0-44b1-a045-9ae36b9bbfb9\") " pod="openshift-ingress-canary/ingress-canary-w6j8h" Jul 07 15:41:06.097129 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.097108 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxqh\" (UniqueName: \"kubernetes.io/projected/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-api-access-qlxqh\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.146556 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.146516 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w6j8h" Jul 07 15:41:06.485194 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.484912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:06.485358 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.485068 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:41:06.485358 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.485235 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcc5bb45d-l9tjf: secret "image-registry-tls" not found Jul 07 15:41:06.485358 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.485299 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls podName:2970c78e-3018-4a3e-8f54-73e26dca206a nodeName:}" failed. No retries permitted until 2026-07-07 15:41:07.485283756 +0000 UTC m=+37.084353018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls") pod "image-registry-5dcc5bb45d-l9tjf" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a") : secret "image-registry-tls" not found Jul 07 15:41:06.586336 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.586287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.586519 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.586356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:06.586519 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.586413 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:06.586519 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:06.586421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:06.586519 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.586487 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm podName:dc0f2445-5253-4dec-abeb-f60ea932d598 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:07.586464437 +0000 UTC m=+37.185533700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4b46w" (UID: "dc0f2445-5253-4dec-abeb-f60ea932d598") : configmap "kube-rbac-proxy" not found Jul 07 15:41:06.586519 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.586520 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:06.586758 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.586561 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls podName:d079eb86-acc0-4b1d-9cb5-873b28271068 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:07.586547504 +0000 UTC m=+37.185616769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls") pod "dns-default-tg7zm" (UID: "d079eb86-acc0-4b1d-9cb5-873b28271068") : secret "dns-default-metrics-tls" not found Jul 07 15:41:06.586758 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.586567 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:06.586758 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:06.586628 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls podName:dc0f2445-5253-4dec-abeb-f60ea932d598 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:07.586611508 +0000 UTC m=+37.185680804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b46w" (UID: "dc0f2445-5253-4dec-abeb-f60ea932d598") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:07.040279 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:07.040252 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w6j8h"] Jul 07 15:41:07.045777 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:07.045752 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5155845_d5c0_44b1_a045_9ae36b9bbfb9.slice/crio-38e86bcb9eb6de02128a8e11d0eb5413452f0d9f076a1eafea2913494e3d90dc WatchSource:0}: Error finding container 38e86bcb9eb6de02128a8e11d0eb5413452f0d9f076a1eafea2913494e3d90dc: Status 404 returned error can't find the container with id 38e86bcb9eb6de02128a8e11d0eb5413452f0d9f076a1eafea2913494e3d90dc Jul 07 15:41:07.495721 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:07.495638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:07.495864 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.495753 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:41:07.495864 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.495765 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcc5bb45d-l9tjf: secret "image-registry-tls" not found Jul 07 15:41:07.495864 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.495814 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls podName:2970c78e-3018-4a3e-8f54-73e26dca206a nodeName:}" failed. No retries permitted until 2026-07-07 15:41:09.495801978 +0000 UTC m=+39.094871239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls") pod "image-registry-5dcc5bb45d-l9tjf" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a") : secret "image-registry-tls" not found Jul 07 15:41:07.596045 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:07.596003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:07.596223 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:07.596061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:07.596223 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:07.596098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:07.596223 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.596122 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:07.596223 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.596188 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm podName:dc0f2445-5253-4dec-abeb-f60ea932d598 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:09.596173826 +0000 UTC m=+39.195243088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4b46w" (UID: "dc0f2445-5253-4dec-abeb-f60ea932d598") : configmap "kube-rbac-proxy" not found Jul 07 15:41:07.596223 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.596199 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:07.596223 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.596203 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:07.596445 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.596240 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls podName:dc0f2445-5253-4dec-abeb-f60ea932d598 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:09.596228754 +0000 UTC m=+39.195298015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b46w" (UID: "dc0f2445-5253-4dec-abeb-f60ea932d598") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:07.596445 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:07.596252 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls podName:d079eb86-acc0-4b1d-9cb5-873b28271068 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:09.596246793 +0000 UTC m=+39.195316055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls") pod "dns-default-tg7zm" (UID: "d079eb86-acc0-4b1d-9cb5-873b28271068") : secret "dns-default-metrics-tls" not found Jul 07 15:41:08.035423 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:08.035385 2576 generic.go:356] "Generic (PLEG): container finished" podID="99d76d9c-ec28-46af-a4a3-cb88c26ec23d" containerID="5e434d3bd883493090b447af4c4cbe2cdcab68bd6ab3f9267b0afa878ac6a6de" exitCode=0 Jul 07 15:41:08.035631 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:08.035469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44mlq" event={"ID":"99d76d9c-ec28-46af-a4a3-cb88c26ec23d","Type":"ContainerDied","Data":"5e434d3bd883493090b447af4c4cbe2cdcab68bd6ab3f9267b0afa878ac6a6de"} Jul 07 15:41:08.036724 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:08.036585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w6j8h" event={"ID":"f5155845-d5c0-44b1-a045-9ae36b9bbfb9","Type":"ContainerStarted","Data":"38e86bcb9eb6de02128a8e11d0eb5413452f0d9f076a1eafea2913494e3d90dc"} Jul 07 15:41:09.043779 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:09.043739 2576 generic.go:356] "Generic (PLEG): container finished" podID="99d76d9c-ec28-46af-a4a3-cb88c26ec23d" containerID="02c5d1022a3cef783a8c087284cea9dd2ada5335476824dfb167326c82280063" exitCode=0 Jul 07 15:41:09.043779 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:09.043783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44mlq" event={"ID":"99d76d9c-ec28-46af-a4a3-cb88c26ec23d","Type":"ContainerDied","Data":"02c5d1022a3cef783a8c087284cea9dd2ada5335476824dfb167326c82280063"} Jul 07 15:41:09.512511 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:09.512489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:09.512642 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:09.512624 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 15:41:09.512678 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:09.512644 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dcc5bb45d-l9tjf: secret "image-registry-tls" not found Jul 07 15:41:09.512711 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:09.512688 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls podName:2970c78e-3018-4a3e-8f54-73e26dca206a nodeName:}" failed. No retries permitted until 2026-07-07 15:41:13.512674695 +0000 UTC m=+43.111743958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls") pod "image-registry-5dcc5bb45d-l9tjf" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a") : secret "image-registry-tls" not found Jul 07 15:41:09.613908 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:09.613882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:09.614060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:09.613935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:09.614060 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:09.613994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:09.614142 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:09.614073 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:09.614142 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:09.614089 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:09.614142 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:09.614140 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls podName:dc0f2445-5253-4dec-abeb-f60ea932d598 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:13.614122926 +0000 UTC m=+43.213192188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b46w" (UID: "dc0f2445-5253-4dec-abeb-f60ea932d598") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:09.614256 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:09.614165 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls podName:d079eb86-acc0-4b1d-9cb5-873b28271068 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:13.614149524 +0000 UTC m=+43.213218787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls") pod "dns-default-tg7zm" (UID: "d079eb86-acc0-4b1d-9cb5-873b28271068") : secret "dns-default-metrics-tls" not found Jul 07 15:41:09.614328 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:09.614310 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0f2445-5253-4dec-abeb-f60ea932d598-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:10.050094 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:10.050046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44mlq" event={"ID":"99d76d9c-ec28-46af-a4a3-cb88c26ec23d","Type":"ContainerStarted","Data":"d03177ca08b459ca03919e059c3524324c0977c7e74df6ab637b0dbcefe057dc"} Jul 07 15:41:10.051961 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:10.051925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w6j8h" event={"ID":"f5155845-d5c0-44b1-a045-9ae36b9bbfb9","Type":"ContainerStarted","Data":"cbcdc81589465495fb91729c606fec931314a83c70707fdf5b50fd4d7a6226b1"} Jul 07 15:41:10.080490 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:10.080427 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-44mlq" podStartSLOduration=19.457329397 podStartE2EDuration="40.080405875s" totalImagesPullingTime="20.623076478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:10.078001862 +0000 UTC m=+39.677071168" watchObservedRunningTime="2026-07-07 15:41:10.080405875 +0000 UTC m=+39.679475161" Jul 07 15:41:10.099172 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:10.099113 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w6j8h" podStartSLOduration=2.236911003 podStartE2EDuration="5.099097069s" totalImagesPullingTime="2.862186066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:10.098683295 +0000 UTC m=+39.697752577" watchObservedRunningTime="2026-07-07 15:41:10.099097069 +0000 UTC m=+39.698166368" Jul 07 15:41:13.544880 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.544833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:13.548997 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.548954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") pod \"image-registry-5dcc5bb45d-l9tjf\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:13.611986 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.611931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:13.645384 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.645349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:13.645502 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.645431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:13.645558 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:13.645498 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:13.645612 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:41:13.645559 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls podName:dc0f2445-5253-4dec-abeb-f60ea932d598 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:21.645542906 +0000 UTC m=+51.244612168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b46w" (UID: "dc0f2445-5253-4dec-abeb-f60ea932d598") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:13.648040 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.648015 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d079eb86-acc0-4b1d-9cb5-873b28271068-metrics-tls\") pod \"dns-default-tg7zm\" (UID: \"d079eb86-acc0-4b1d-9cb5-873b28271068\") " pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:13.659924 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.659889 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:13.778439 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.778408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dcc5bb45d-l9tjf"] Jul 07 15:41:13.781219 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:13.781182 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2970c78e_3018_4a3e_8f54_73e26dca206a.slice/crio-0f1583f29506b8a7e6bbdff6b9e0902b33aa5327ce6e00b433df4ba7ed86131c WatchSource:0}: Error finding container 0f1583f29506b8a7e6bbdff6b9e0902b33aa5327ce6e00b433df4ba7ed86131c: Status 404 returned error can't find the container with id 0f1583f29506b8a7e6bbdff6b9e0902b33aa5327ce6e00b433df4ba7ed86131c Jul 07 15:41:13.817813 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:13.817786 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tg7zm"] Jul 07 15:41:13.820515 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:13.820483 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd079eb86_acc0_4b1d_9cb5_873b28271068.slice/crio-31fc5966508f76c4beaea2f7fdaa619b74f5ac0ad195efc8bac83a8bf54aafb8 WatchSource:0}: Error finding container 31fc5966508f76c4beaea2f7fdaa619b74f5ac0ad195efc8bac83a8bf54aafb8: Status 404 returned error can't find the container with id 31fc5966508f76c4beaea2f7fdaa619b74f5ac0ad195efc8bac83a8bf54aafb8 Jul 07 15:41:14.063600 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:14.063561 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" event={"ID":"2970c78e-3018-4a3e-8f54-73e26dca206a","Type":"ContainerStarted","Data":"f85712c25b20eeda54320bf52c829e097e1532476a9c228a60922d7c2ed9d9f3"} Jul 07 15:41:14.063788 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:14.063630 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:14.063788 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:14.063654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" event={"ID":"2970c78e-3018-4a3e-8f54-73e26dca206a","Type":"ContainerStarted","Data":"0f1583f29506b8a7e6bbdff6b9e0902b33aa5327ce6e00b433df4ba7ed86131c"} Jul 07 15:41:14.064658 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:14.064634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg7zm" event={"ID":"d079eb86-acc0-4b1d-9cb5-873b28271068","Type":"ContainerStarted","Data":"31fc5966508f76c4beaea2f7fdaa619b74f5ac0ad195efc8bac83a8bf54aafb8"} Jul 07 15:41:14.090268 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:14.090166 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" podStartSLOduration=54.090151085 podStartE2EDuration="54.090151085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:14.089332009 +0000 UTC m=+43.688401317" watchObservedRunningTime="2026-07-07 15:41:14.090151085 +0000 UTC m=+43.689220368" Jul 07 15:41:17.075082 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:17.075046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg7zm" event={"ID":"d079eb86-acc0-4b1d-9cb5-873b28271068","Type":"ContainerStarted","Data":"5b23a3ae59cf23d9f73c52ba23d30fe10e1c931fa8f0ca49f772c327479b24cb"} Jul 07 15:41:17.075632 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:17.075100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tg7zm" event={"ID":"d079eb86-acc0-4b1d-9cb5-873b28271068","Type":"ContainerStarted","Data":"32eeb74665ad577856dc48507aeed8ff35160b91f2382fa26143c9a4bba2e414"} Jul 07 15:41:17.075632 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:17.075170 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:17.095822 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:17.095778 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tg7zm" podStartSLOduration=9.896319182 podStartE2EDuration="12.095759054s" totalImagesPullingTime="2.199439872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:17.09554781 +0000 UTC m=+46.694617072" watchObservedRunningTime="2026-07-07 15:41:17.095759054 +0000 UTC m=+46.694828338" Jul 07 15:41:21.707634 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:21.707604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:21.710041 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:21.710019 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0f2445-5253-4dec-abeb-f60ea932d598-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b46w\" (UID: \"dc0f2445-5253-4dec-abeb-f60ea932d598\") " pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:21.754362 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:21.754332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4b46w" Jul 07 15:41:21.890893 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:21.890865 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4b46w"] Jul 07 15:41:21.894072 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:21.894043 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0f2445_5253_4dec_abeb_f60ea932d598.slice/crio-79baa6bef41b9a50cfa0118f4070853412b817a60699f0178dfd60778420b5cb WatchSource:0}: Error finding container 79baa6bef41b9a50cfa0118f4070853412b817a60699f0178dfd60778420b5cb: Status 404 returned error can't find the container with id 79baa6bef41b9a50cfa0118f4070853412b817a60699f0178dfd60778420b5cb Jul 07 15:41:22.088176 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:22.088147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b46w" event={"ID":"dc0f2445-5253-4dec-abeb-f60ea932d598","Type":"ContainerStarted","Data":"79baa6bef41b9a50cfa0118f4070853412b817a60699f0178dfd60778420b5cb"} Jul 07 15:41:26.967884 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.967846 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk"] Jul 07 15:41:26.970928 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.970911 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz"] Jul 07 15:41:26.971087 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.971070 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:26.974016 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.973996 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:26.975651 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.975627 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 15:41:26.975797 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.975636 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 15:41:26.975905 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.975739 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 15:41:26.976754 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.976736 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69"] Jul 07 15:41:26.978254 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.978125 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 15:41:26.978511 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.978295 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 15:41:26.978727 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.978707 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 15:41:26.978869 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.978827 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 15:41:26.979038 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.979021 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 15:41:26.979692 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.979675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" Jul 07 15:41:26.982160 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.982141 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 15:41:26.982270 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.982254 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-gqpn4\"" Jul 07 15:41:26.987502 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.987472 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz"] Jul 07 15:41:26.988356 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.988339 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dcc5bb45d-l9tjf"] Jul 07 15:41:26.992221 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.990930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk"] Jul 07 15:41:26.996915 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:26.996894 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69"] Jul 07 15:41:27.051476 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.051642 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-hub\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.051642 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/013a97c4-ec99-457d-967a-980fbfa345e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.051760 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbrg\" (UniqueName: \"kubernetes.io/projected/013a97c4-ec99-457d-967a-980fbfa345e7-kube-api-access-ddbrg\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.051760 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/013a97c4-ec99-457d-967a-980fbfa345e7-tmp\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.051760 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ec24888-8fa6-41a7-858e-875c2267aa40-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84b9867794-h8w69\" (UID: \"2ec24888-8fa6-41a7-858e-875c2267aa40\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" Jul 07 15:41:27.051897 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6657931c-aa43-4a6a-a548-6e76c7ee961a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.051897 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.052029 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.051957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbgw\" (UniqueName: \"kubernetes.io/projected/2ec24888-8fa6-41a7-858e-875c2267aa40-kube-api-access-6cbgw\") pod \"managed-serviceaccount-addon-agent-84b9867794-h8w69\" (UID: \"2ec24888-8fa6-41a7-858e-875c2267aa40\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" Jul 07 15:41:27.052094 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.052071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-ca\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.052144 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.052112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqp4\" (UniqueName: \"kubernetes.io/projected/6657931c-aa43-4a6a-a548-6e76c7ee961a-kube-api-access-bgqp4\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.064956 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.064932 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b8b55b46c-5cpb4"] Jul 07 15:41:27.067915 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.067899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.079795 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.079772 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tg7zm" Jul 07 15:41:27.087061 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.087042 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b8b55b46c-5cpb4"] Jul 07 15:41:27.102357 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.102331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b46w" event={"ID":"dc0f2445-5253-4dec-abeb-f60ea932d598","Type":"ContainerStarted","Data":"9640d7d5d5a84221fb284eb4847f16fa82973b32eaaad8652dac50d1b684de6e"} Jul 07 15:41:27.152567 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ec24888-8fa6-41a7-858e-875c2267aa40-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84b9867794-h8w69\" (UID: \"2ec24888-8fa6-41a7-858e-875c2267aa40\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" Jul 07 15:41:27.152715 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692wm\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-kube-api-access-692wm\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.152715 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6657931c-aa43-4a6a-a548-6e76c7ee961a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.152715 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3258b7f5-c7b1-45df-94b7-da0f8af0e645-trusted-ca\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.152715 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.152913 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbgw\" (UniqueName: \"kubernetes.io/projected/2ec24888-8fa6-41a7-858e-875c2267aa40-kube-api-access-6cbgw\") pod \"managed-serviceaccount-addon-agent-84b9867794-h8w69\" (UID: \"2ec24888-8fa6-41a7-858e-875c2267aa40\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" Jul 07 15:41:27.152913 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3258b7f5-c7b1-45df-94b7-da0f8af0e645-registry-certificates\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.152913 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3258b7f5-c7b1-45df-94b7-da0f8af0e645-installation-pull-secrets\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.152913 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-registry-tls\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.152913 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-ca\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.153205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.152951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp4\" (UniqueName: \"kubernetes.io/projected/6657931c-aa43-4a6a-a548-6e76c7ee961a-kube-api-access-bgqp4\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.153205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.153205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-bound-sa-token\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.153205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3258b7f5-c7b1-45df-94b7-da0f8af0e645-image-registry-private-configuration\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.153205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3258b7f5-c7b1-45df-94b7-da0f8af0e645-ca-trust-extracted\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.153205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-hub\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.153205 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/013a97c4-ec99-457d-967a-980fbfa345e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.153462 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbrg\" (UniqueName: \"kubernetes.io/projected/013a97c4-ec99-457d-967a-980fbfa345e7-kube-api-access-ddbrg\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.153462 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/013a97c4-ec99-457d-967a-980fbfa345e7-tmp\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.153606 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.153588 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/013a97c4-ec99-457d-967a-980fbfa345e7-tmp\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.154075 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.154038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6657931c-aa43-4a6a-a548-6e76c7ee961a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.156244 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.156220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ec24888-8fa6-41a7-858e-875c2267aa40-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84b9867794-h8w69\" (UID: \"2ec24888-8fa6-41a7-858e-875c2267aa40\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" Jul 07 15:41:27.156358 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.156225 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-ca\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.157343 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.157318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.157948 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.157911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.158565 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.158529 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/013a97c4-ec99-457d-967a-980fbfa345e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.158774 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.158754 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6657931c-aa43-4a6a-a548-6e76c7ee961a-hub\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.169018 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.168998 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbgw\" (UniqueName: \"kubernetes.io/projected/2ec24888-8fa6-41a7-858e-875c2267aa40-kube-api-access-6cbgw\") pod \"managed-serviceaccount-addon-agent-84b9867794-h8w69\" (UID: \"2ec24888-8fa6-41a7-858e-875c2267aa40\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" Jul 07 15:41:27.176882 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.176842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqp4\" (UniqueName: \"kubernetes.io/projected/6657931c-aa43-4a6a-a548-6e76c7ee961a-kube-api-access-bgqp4\") pod \"cluster-proxy-proxy-agent-d5b876f75-ktpzk\" (UID: \"6657931c-aa43-4a6a-a548-6e76c7ee961a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.177758 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.177706 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbrg\" (UniqueName: \"kubernetes.io/projected/013a97c4-ec99-457d-967a-980fbfa345e7-kube-api-access-ddbrg\") pod \"klusterlet-addon-workmgr-686ccf8ccd-w27hz\" (UID: \"013a97c4-ec99-457d-967a-980fbfa345e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.254094 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3258b7f5-c7b1-45df-94b7-da0f8af0e645-trusted-ca\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.254208 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3258b7f5-c7b1-45df-94b7-da0f8af0e645-registry-certificates\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.254208 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3258b7f5-c7b1-45df-94b7-da0f8af0e645-installation-pull-secrets\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.254208 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-registry-tls\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.254359 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-bound-sa-token\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.254359 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3258b7f5-c7b1-45df-94b7-da0f8af0e645-image-registry-private-configuration\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.254359 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3258b7f5-c7b1-45df-94b7-da0f8af0e645-ca-trust-extracted\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.254509 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692wm\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-kube-api-access-692wm\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.254856 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254793 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3258b7f5-c7b1-45df-94b7-da0f8af0e645-ca-trust-extracted\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.255088 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254891 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3258b7f5-c7b1-45df-94b7-da0f8af0e645-trusted-ca\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.255088 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.254927 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3258b7f5-c7b1-45df-94b7-da0f8af0e645-registry-certificates\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.256471 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.256443 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3258b7f5-c7b1-45df-94b7-da0f8af0e645-installation-pull-secrets\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.256550 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.256540 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3258b7f5-c7b1-45df-94b7-da0f8af0e645-image-registry-private-configuration\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.256758 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.256740 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-registry-tls\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.265939 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.265918 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-bound-sa-token\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.266262 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.266244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692wm\" (UniqueName: \"kubernetes.io/projected/3258b7f5-c7b1-45df-94b7-da0f8af0e645-kube-api-access-692wm\") pod \"image-registry-b8b55b46c-5cpb4\" (UID: \"3258b7f5-c7b1-45df-94b7-da0f8af0e645\") " pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.295941 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.295917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" Jul 07 15:41:27.302688 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.302671 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:27.307639 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.307623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" Jul 07 15:41:27.390362 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.390332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:27.451421 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.451352 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk"] Jul 07 15:41:27.465756 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.465685 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz"] Jul 07 15:41:27.494207 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.494183 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69"] Jul 07 15:41:27.497879 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:27.497854 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec24888_8fa6_41a7_858e_875c2267aa40.slice/crio-8e86c972d5a20e82c88886bf72d44bca50b05b4b53d201488b917511bb63800a WatchSource:0}: Error finding container 8e86c972d5a20e82c88886bf72d44bca50b05b4b53d201488b917511bb63800a: Status 404 returned error can't find the container with id 8e86c972d5a20e82c88886bf72d44bca50b05b4b53d201488b917511bb63800a Jul 07 15:41:27.557454 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:27.557422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b8b55b46c-5cpb4"] Jul 07 15:41:27.562349 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:27.562320 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3258b7f5_c7b1_45df_94b7_da0f8af0e645.slice/crio-1f1cc8d4c7245af60a47ecb3ec786d207c9d7b16b8a8aca59725225f3f25b2f1 WatchSource:0}: Error finding container 1f1cc8d4c7245af60a47ecb3ec786d207c9d7b16b8a8aca59725225f3f25b2f1: Status 404 returned error can't find the container with id 1f1cc8d4c7245af60a47ecb3ec786d207c9d7b16b8a8aca59725225f3f25b2f1 Jul 07 15:41:28.108389 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:28.108264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" event={"ID":"3258b7f5-c7b1-45df-94b7-da0f8af0e645","Type":"ContainerStarted","Data":"248d9b97b44f03735f8e08365220f8982cccbb3aa7922cb501f9ca73c9d77ecc"} Jul 07 15:41:28.108389 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:28.108323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" event={"ID":"3258b7f5-c7b1-45df-94b7-da0f8af0e645","Type":"ContainerStarted","Data":"1f1cc8d4c7245af60a47ecb3ec786d207c9d7b16b8a8aca59725225f3f25b2f1"} Jul 07 15:41:28.108916 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:28.108523 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:28.109732 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:28.109702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" event={"ID":"2ec24888-8fa6-41a7-858e-875c2267aa40","Type":"ContainerStarted","Data":"8e86c972d5a20e82c88886bf72d44bca50b05b4b53d201488b917511bb63800a"} Jul 07 15:41:28.111920 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:28.111853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b46w" event={"ID":"dc0f2445-5253-4dec-abeb-f60ea932d598","Type":"ContainerStarted","Data":"13878ed30e4e7ae6ef0b70d0b336db966ebd7956056c33adf11ee0e6e3369cbb"} Jul 07 15:41:28.113312 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:28.113259 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" event={"ID":"6657931c-aa43-4a6a-a548-6e76c7ee961a","Type":"ContainerStarted","Data":"b661687c0040b64024ff6cd368f4406390e37cbd870214c219a28941961b2f7f"} Jul 07 15:41:28.115389 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:28.115365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" event={"ID":"013a97c4-ec99-457d-967a-980fbfa345e7","Type":"ContainerStarted","Data":"6f450808bdcc87e50bf14272379f73b701b366ca5575a57032e48083d71dd133"} Jul 07 15:41:28.141557 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:28.138772 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" podStartSLOduration=1.138752778 podStartE2EDuration="1.138752778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:28.13488392 +0000 UTC m=+57.733953206" watchObservedRunningTime="2026-07-07 15:41:28.138752778 +0000 UTC m=+57.737822063" Jul 07 15:41:32.025556 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:32.025530 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8p85w" Jul 07 15:41:35.146474 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.146442 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" event={"ID":"2ec24888-8fa6-41a7-858e-875c2267aa40","Type":"ContainerStarted","Data":"7a7074582bdec36a7787a38457ea32ee29ed51b79fc3012bd14df1030fd13979"} Jul 07 15:41:35.148521 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.148486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b46w" event={"ID":"dc0f2445-5253-4dec-abeb-f60ea932d598","Type":"ContainerStarted","Data":"91cb40c1339d3f94a120710e7f1ccb7cbb9a3197c999efda6d77f06102f8de28"} Jul 07 15:41:35.150783 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.150759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" event={"ID":"6657931c-aa43-4a6a-a548-6e76c7ee961a","Type":"ContainerStarted","Data":"29a0033e6b27333c003033e41e81f2b9fa5b0b6b56f2f4186aa136060901174e"} Jul 07 15:41:35.150871 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.150797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" event={"ID":"6657931c-aa43-4a6a-a548-6e76c7ee961a","Type":"ContainerStarted","Data":"9c875f3955d852bc3e7ba30dfff94567fc068f8e4b8030ed398a2fa4c7ab350a"} Jul 07 15:41:35.150871 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.150815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" event={"ID":"6657931c-aa43-4a6a-a548-6e76c7ee961a","Type":"ContainerStarted","Data":"0cb525b1ec9efd7b31a24c5f97cc7cef0a9495c2815320f17baa7da6591a5538"} Jul 07 15:41:35.152085 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.152065 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" event={"ID":"013a97c4-ec99-457d-967a-980fbfa345e7","Type":"ContainerStarted","Data":"54e461a89c2bd37e6f20d0d7f70d27bfbf23497fe4e6e14963c9fe48632140e4"} Jul 07 15:41:35.152278 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.152264 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:35.153727 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.153709 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" Jul 07 15:41:35.164695 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.164658 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84b9867794-h8w69" podStartSLOduration=1.937751719 podStartE2EDuration="9.164646471s" totalImagesPullingTime="7.226894752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:35.164219989 +0000 UTC m=+64.763289266" watchObservedRunningTime="2026-07-07 15:41:35.164646471 +0000 UTC m=+64.763715732" Jul 07 15:41:35.185551 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.185509 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5b876f75-ktpzk" podStartSLOduration=1.944327742 podStartE2EDuration="9.18549719s" totalImagesPullingTime="7.241169448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:35.18456909 +0000 UTC m=+64.783638374" watchObservedRunningTime="2026-07-07 15:41:35.18549719 +0000 UTC m=+64.784566474" Jul 07 15:41:35.208260 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.208181 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4b46w" podStartSLOduration=17.491535831 podStartE2EDuration="30.208168751s" totalImagesPullingTime="12.71663292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:35.207400116 +0000 UTC m=+64.806469400" watchObservedRunningTime="2026-07-07 15:41:35.208168751 +0000 UTC m=+64.807238035" Jul 07 15:41:35.230691 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:35.230653 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686ccf8ccd-w27hz" podStartSLOduration=1.974686154 podStartE2EDuration="9.230641277s" totalImagesPullingTime="7.255955123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:35.23041549 +0000 UTC m=+64.829484777" watchObservedRunningTime="2026-07-07 15:41:35.230641277 +0000 UTC m=+64.829710573" Jul 07 15:41:36.639820 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.639782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:41:36.640273 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.639833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:41:36.640273 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.639852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:41:36.642717 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.642700 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.643989 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.643957 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 15:41:36.644100 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.644014 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 15:41:36.652751 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.652722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27cbb2a7-11c0-43e1-9d8f-1f7e630465d1-original-pull-secret\") pod \"global-pull-secret-syncer-cvlj7\" (UID: \"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1\") " pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:41:36.652751 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.652745 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/500054cf-33d6-4dc7-9dd7-272bbacd6b95-metrics-certs\") pod \"network-metrics-daemon-27cjl\" (UID: \"500054cf-33d6-4dc7-9dd7-272bbacd6b95\") " pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:41:36.653185 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.653168 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.662998 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.662958 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftjh\" (UniqueName: \"kubernetes.io/projected/a93cf57e-bd7f-4a64-9a95-d840b2877541-kube-api-access-5ftjh\") pod \"network-check-target-ltsg9\" (UID: \"a93cf57e-bd7f-4a64-9a95-d840b2877541\") " pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:41:36.726242 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.726222 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jh4tr\"" Jul 07 15:41:36.733548 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.733534 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27cjl" Jul 07 15:41:36.733664 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.733648 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvlj7" Jul 07 15:41:36.743081 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.743061 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v8g69\"" Jul 07 15:41:36.750796 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.750761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:41:36.885058 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.885026 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-27cjl"] Jul 07 15:41:36.889877 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:36.889810 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500054cf_33d6_4dc7_9dd7_272bbacd6b95.slice/crio-eded918562628facad39e876ea62fd1c9bee31ac078c4af1144fe4c9a3d5ad84 WatchSource:0}: Error finding container eded918562628facad39e876ea62fd1c9bee31ac078c4af1144fe4c9a3d5ad84: Status 404 returned error can't find the container with id eded918562628facad39e876ea62fd1c9bee31ac078c4af1144fe4c9a3d5ad84 Jul 07 15:41:36.902016 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.901870 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cvlj7"] Jul 07 15:41:36.904010 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:36.903984 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cbb2a7_11c0_43e1_9d8f_1f7e630465d1.slice/crio-11172aaf8d1340b17b11572fedfec7c4fd29ce63ca4080ab44aacc1715cbb968 WatchSource:0}: Error finding container 11172aaf8d1340b17b11572fedfec7c4fd29ce63ca4080ab44aacc1715cbb968: Status 404 returned error can't find the container with id 11172aaf8d1340b17b11572fedfec7c4fd29ce63ca4080ab44aacc1715cbb968 Jul 07 15:41:36.921031 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.921014 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ltsg9"] Jul 07 15:41:36.923049 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:36.923030 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93cf57e_bd7f_4a64_9a95_d840b2877541.slice/crio-681ab74aca64c9a335a014240008a198deeaaec6afe4961b691c471653344689 WatchSource:0}: Error finding container 681ab74aca64c9a335a014240008a198deeaaec6afe4961b691c471653344689: Status 404 returned error can't find the container with id 681ab74aca64c9a335a014240008a198deeaaec6afe4961b691c471653344689 Jul 07 15:41:36.993600 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:36.993571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:37.159549 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:37.159459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ltsg9" event={"ID":"a93cf57e-bd7f-4a64-9a95-d840b2877541","Type":"ContainerStarted","Data":"681ab74aca64c9a335a014240008a198deeaaec6afe4961b691c471653344689"} Jul 07 15:41:37.160689 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:37.160667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-27cjl" event={"ID":"500054cf-33d6-4dc7-9dd7-272bbacd6b95","Type":"ContainerStarted","Data":"eded918562628facad39e876ea62fd1c9bee31ac078c4af1144fe4c9a3d5ad84"} Jul 07 15:41:37.161770 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:37.161750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cvlj7" event={"ID":"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1","Type":"ContainerStarted","Data":"11172aaf8d1340b17b11572fedfec7c4fd29ce63ca4080ab44aacc1715cbb968"} Jul 07 15:41:39.174291 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:39.174198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-27cjl" event={"ID":"500054cf-33d6-4dc7-9dd7-272bbacd6b95","Type":"ContainerStarted","Data":"33420bb36c3107261c1583f9a470e6419caa73317a025157219b188a9d1283f0"} Jul 07 15:41:39.174291 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:39.174265 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-27cjl" event={"ID":"500054cf-33d6-4dc7-9dd7-272bbacd6b95","Type":"ContainerStarted","Data":"eee318d4dc50d5acae186a84cef957e4669cc81d6bd7ef19aee55964ba4294ab"} Jul 07 15:41:43.190931 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.190894 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ltsg9" event={"ID":"a93cf57e-bd7f-4a64-9a95-d840b2877541","Type":"ContainerStarted","Data":"02eb87ee953cc182d4fee2728d893a8dd3bf9cf61345da2132e046c6a0a79668"} Jul 07 15:41:43.191398 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.190962 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:41:43.192325 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.192301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cvlj7" event={"ID":"27cbb2a7-11c0-43e1-9d8f-1f7e630465d1","Type":"ContainerStarted","Data":"050cf029741582ee435fdd966226ae6842d3f81863b713d5df985da0a7ebcbfb"} Jul 07 15:41:43.208727 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.208676 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-27cjl" podStartSLOduration=70.490780755 podStartE2EDuration="1m12.208664131s" totalImagesPullingTime="1.717883376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:39.19791235 +0000 UTC m=+68.796981660" watchObservedRunningTime="2026-07-07 15:41:43.208664131 +0000 UTC m=+72.807733415" Jul 07 15:41:43.209250 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.209227 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ltsg9" podStartSLOduration=67.972648723 podStartE2EDuration="1m13.209219619s" totalImagesPullingTime="5.236570896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:43.208380589 +0000 UTC m=+72.807449874" watchObservedRunningTime="2026-07-07 15:41:43.209219619 +0000 UTC m=+72.808288902" Jul 07 15:41:43.226790 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.226745 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cvlj7" podStartSLOduration=66.965356857 podStartE2EDuration="1m12.226730301s" totalImagesPullingTime="5.261373444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:43.226139364 +0000 UTC m=+72.825208649" watchObservedRunningTime="2026-07-07 15:41:43.226730301 +0000 UTC m=+72.825799585" Jul 07 15:41:43.874445 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.874409 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dcsnk"] Jul 07 15:41:43.877637 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.877619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:43.880501 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.880478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 15:41:43.880662 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.880645 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 15:41:43.881012 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.880998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 15:41:43.881231 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.881212 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 15:41:43.882033 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.882018 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zws6j\"" Jul 07 15:41:43.882113 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.882049 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 15:41:43.882156 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:43.882057 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 15:41:44.000735 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.000695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-tls\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.000735 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.000731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1861a98-2c66-4753-a243-15132f84c77b-metrics-client-ca\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.001023 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.000756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.001023 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.000812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-textfile\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.001023 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.000864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-wtmp\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.001023 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.000898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-root\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.001023 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.000921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.001023 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.001008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-sys\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.001232 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.001050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9zs\" (UniqueName: \"kubernetes.io/projected/a1861a98-2c66-4753-a243-15132f84c77b-kube-api-access-tg9zs\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101649 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-sys\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101649 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9zs\" (UniqueName: \"kubernetes.io/projected/a1861a98-2c66-4753-a243-15132f84c77b-kube-api-access-tg9zs\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101892 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-tls\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101892 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1861a98-2c66-4753-a243-15132f84c77b-metrics-client-ca\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101892 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101719 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-sys\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101892 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101892 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-textfile\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101892 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-wtmp\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101892 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-root\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.101892 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.102236 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.101931 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-root\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.102236 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.102014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-wtmp\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.102299 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.102245 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-textfile\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.102413 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.102395 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1861a98-2c66-4753-a243-15132f84c77b-metrics-client-ca\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.102455 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.102427 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.104348 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.104330 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.104424 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.104408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1861a98-2c66-4753-a243-15132f84c77b-node-exporter-tls\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.111525 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.111497 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9zs\" (UniqueName: \"kubernetes.io/projected/a1861a98-2c66-4753-a243-15132f84c77b-kube-api-access-tg9zs\") pod \"node-exporter-dcsnk\" (UID: \"a1861a98-2c66-4753-a243-15132f84c77b\") " pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.186834 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:44.186752 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dcsnk" Jul 07 15:41:44.199178 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:44.199149 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1861a98_2c66_4753_a243_15132f84c77b.slice/crio-a21a163bc7971766c187782407b4bd10742d4dbd75f73dbf30217252de5b372d WatchSource:0}: Error finding container a21a163bc7971766c187782407b4bd10742d4dbd75f73dbf30217252de5b372d: Status 404 returned error can't find the container with id a21a163bc7971766c187782407b4bd10742d4dbd75f73dbf30217252de5b372d Jul 07 15:41:45.200524 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:45.200489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcsnk" event={"ID":"a1861a98-2c66-4753-a243-15132f84c77b","Type":"ContainerStarted","Data":"a21a163bc7971766c187782407b4bd10742d4dbd75f73dbf30217252de5b372d"} Jul 07 15:41:46.206284 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:46.206248 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1861a98-2c66-4753-a243-15132f84c77b" containerID="1061147bd475204452e0a272426cf72e605aed7fc523a3ac2265dc38de5b4747" exitCode=0 Jul 07 15:41:46.206669 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:46.206312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcsnk" event={"ID":"a1861a98-2c66-4753-a243-15132f84c77b","Type":"ContainerDied","Data":"1061147bd475204452e0a272426cf72e605aed7fc523a3ac2265dc38de5b4747"} Jul 07 15:41:47.212171 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:47.212137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcsnk" event={"ID":"a1861a98-2c66-4753-a243-15132f84c77b","Type":"ContainerStarted","Data":"25cfe467caf8a7d63496742e6bf4a20ff3be7534ec7e6d21cd356754c53548d7"} Jul 07 15:41:47.212575 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:47.212194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcsnk" event={"ID":"a1861a98-2c66-4753-a243-15132f84c77b","Type":"ContainerStarted","Data":"14741b2734b448595b2bf416a9e6130efdf7e383ace3c25ec3e4d5cde6de25a6"} Jul 07 15:41:49.124844 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:49.124810 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b8b55b46c-5cpb4" Jul 07 15:41:49.150336 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:49.150285 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dcsnk" podStartSLOduration=5.067051542 podStartE2EDuration="6.150268808s" totalImagesPullingTime="1.012860293s" totalInitContainerRuntime="70.356973ms" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:47.23618846 +0000 UTC m=+76.835257744" watchObservedRunningTime="2026-07-07 15:41:49.150268808 +0000 UTC m=+78.749338092" Jul 07 15:41:52.011958 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.011830 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" podUID="2970c78e-3018-4a3e-8f54-73e26dca206a" containerName="registry" containerID="cri-o://f85712c25b20eeda54320bf52c829e097e1532476a9c228a60922d7c2ed9d9f3" gracePeriod=30 Jul 07 15:41:52.230076 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.230046 2576 generic.go:356] "Generic (PLEG): container finished" podID="2970c78e-3018-4a3e-8f54-73e26dca206a" containerID="f85712c25b20eeda54320bf52c829e097e1532476a9c228a60922d7c2ed9d9f3" exitCode=0 Jul 07 15:41:52.230229 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.230120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" event={"ID":"2970c78e-3018-4a3e-8f54-73e26dca206a","Type":"ContainerDied","Data":"f85712c25b20eeda54320bf52c829e097e1532476a9c228a60922d7c2ed9d9f3"} Jul 07 15:41:52.257165 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.257144 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:52.366817 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.366785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-bound-sa-token\") pod \"2970c78e-3018-4a3e-8f54-73e26dca206a\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " Jul 07 15:41:52.367015 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.366830 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-installation-pull-secrets\") pod \"2970c78e-3018-4a3e-8f54-73e26dca206a\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " Jul 07 15:41:52.367015 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.366851 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6rbd\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-kube-api-access-q6rbd\") pod \"2970c78e-3018-4a3e-8f54-73e26dca206a\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " Jul 07 15:41:52.367015 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.366894 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") pod \"2970c78e-3018-4a3e-8f54-73e26dca206a\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " Jul 07 15:41:52.367015 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.366947 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-certificates\") pod \"2970c78e-3018-4a3e-8f54-73e26dca206a\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " Jul 07 15:41:52.367015 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.366999 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-image-registry-private-configuration\") pod \"2970c78e-3018-4a3e-8f54-73e26dca206a\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " Jul 07 15:41:52.367249 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.367029 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2970c78e-3018-4a3e-8f54-73e26dca206a-ca-trust-extracted\") pod \"2970c78e-3018-4a3e-8f54-73e26dca206a\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " Jul 07 15:41:52.367249 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.367072 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-trusted-ca\") pod \"2970c78e-3018-4a3e-8f54-73e26dca206a\" (UID: \"2970c78e-3018-4a3e-8f54-73e26dca206a\") " Jul 07 15:41:52.367633 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.367566 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2970c78e-3018-4a3e-8f54-73e26dca206a" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:41:52.367804 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.367780 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2970c78e-3018-4a3e-8f54-73e26dca206a" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:41:52.369653 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.369619 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2970c78e-3018-4a3e-8f54-73e26dca206a" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:41:52.369769 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.369698 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2970c78e-3018-4a3e-8f54-73e26dca206a" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:41:52.369983 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.369941 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2970c78e-3018-4a3e-8f54-73e26dca206a" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:41:52.370091 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.370006 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-kube-api-access-q6rbd" (OuterVolumeSpecName: "kube-api-access-q6rbd") pod "2970c78e-3018-4a3e-8f54-73e26dca206a" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a"). InnerVolumeSpecName "kube-api-access-q6rbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:41:52.370091 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.370061 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2970c78e-3018-4a3e-8f54-73e26dca206a" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:41:52.376644 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.376599 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2970c78e-3018-4a3e-8f54-73e26dca206a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2970c78e-3018-4a3e-8f54-73e26dca206a" (UID: "2970c78e-3018-4a3e-8f54-73e26dca206a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:41:52.467833 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.467798 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-tls\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:41:52.467833 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.467828 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-registry-certificates\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:41:52.467833 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.467839 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-image-registry-private-configuration\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:41:52.468093 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.467848 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2970c78e-3018-4a3e-8f54-73e26dca206a-ca-trust-extracted\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:41:52.468093 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.467856 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2970c78e-3018-4a3e-8f54-73e26dca206a-trusted-ca\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:41:52.468093 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.467865 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-bound-sa-token\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:41:52.468093 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.467872 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2970c78e-3018-4a3e-8f54-73e26dca206a-installation-pull-secrets\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:41:52.468093 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:52.467879 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6rbd\" (UniqueName: \"kubernetes.io/projected/2970c78e-3018-4a3e-8f54-73e26dca206a-kube-api-access-q6rbd\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:41:53.235063 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:53.235024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" event={"ID":"2970c78e-3018-4a3e-8f54-73e26dca206a","Type":"ContainerDied","Data":"0f1583f29506b8a7e6bbdff6b9e0902b33aa5327ce6e00b433df4ba7ed86131c"} Jul 07 15:41:53.235534 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:53.235088 2576 scope.go:117] "RemoveContainer" containerID="f85712c25b20eeda54320bf52c829e097e1532476a9c228a60922d7c2ed9d9f3" Jul 07 15:41:53.235534 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:53.235037 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dcc5bb45d-l9tjf" Jul 07 15:41:53.257196 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:53.257170 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dcc5bb45d-l9tjf"] Jul 07 15:41:53.260790 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:53.260770 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dcc5bb45d-l9tjf"] Jul 07 15:41:54.805183 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:54.805152 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2970c78e-3018-4a3e-8f54-73e26dca206a" path="/var/lib/kubelet/pods/2970c78e-3018-4a3e-8f54-73e26dca206a/volumes" Jul 07 15:41:57.718112 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.718072 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78f79c85b7-t6gjn"] Jul 07 15:41:57.718633 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.718376 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2970c78e-3018-4a3e-8f54-73e26dca206a" containerName="registry" Jul 07 15:41:57.718633 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.718393 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2970c78e-3018-4a3e-8f54-73e26dca206a" containerName="registry" Jul 07 15:41:57.718633 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.718474 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2970c78e-3018-4a3e-8f54-73e26dca206a" containerName="registry" Jul 07 15:41:57.795342 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.795304 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78f79c85b7-t6gjn"] Jul 07 15:41:57.795498 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.795421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:57.798993 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.798953 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 15:41:57.799145 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.799127 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 15:41:57.800093 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.800069 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 15:41:57.800093 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.800084 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 15:41:57.800349 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.800334 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 15:41:57.800452 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.800430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 15:41:57.800635 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.800618 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t87sb\"" Jul 07 15:41:57.800722 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.800663 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 15:41:57.805382 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.805365 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 15:41:57.903751 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.903719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwn7l\" (UniqueName: \"kubernetes.io/projected/d8b67002-9055-48bb-a9e9-637b820659ac-kube-api-access-mwn7l\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:57.903751 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.903752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-serving-cert\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:57.904013 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.903773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-trusted-ca-bundle\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:57.904013 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.903796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-oauth-serving-cert\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:57.904013 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.903848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-service-ca\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:57.904013 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.903874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-console-config\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:57.904013 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:57.903898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-oauth-config\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.004964 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.004872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-oauth-config\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.004964 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.004911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwn7l\" (UniqueName: \"kubernetes.io/projected/d8b67002-9055-48bb-a9e9-637b820659ac-kube-api-access-mwn7l\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.004964 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.004939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-serving-cert\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.004964 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.004963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-trusted-ca-bundle\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.005344 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.005021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-oauth-serving-cert\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.005344 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.005066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-service-ca\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.005344 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.005104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-console-config\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.005825 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.005803 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-oauth-serving-cert\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.005914 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.005811 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-console-config\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.006155 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.006134 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-service-ca\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.006290 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.006269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-trusted-ca-bundle\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.007638 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.007614 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-serving-cert\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.007720 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.007653 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-oauth-config\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.013929 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.013911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwn7l\" (UniqueName: \"kubernetes.io/projected/d8b67002-9055-48bb-a9e9-637b820659ac-kube-api-access-mwn7l\") pod \"console-78f79c85b7-t6gjn\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.104669 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.104629 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:41:58.289030 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:58.286408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78f79c85b7-t6gjn"] Jul 07 15:41:58.290403 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:41:58.290369 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b67002_9055_48bb_a9e9_637b820659ac.slice/crio-1659b7a0a524ba1ff329df159f0670f6319fd06ca8116817c423389e19a951cc WatchSource:0}: Error finding container 1659b7a0a524ba1ff329df159f0670f6319fd06ca8116817c423389e19a951cc: Status 404 returned error can't find the container with id 1659b7a0a524ba1ff329df159f0670f6319fd06ca8116817c423389e19a951cc Jul 07 15:41:59.261074 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:41:59.261003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f79c85b7-t6gjn" event={"ID":"d8b67002-9055-48bb-a9e9-637b820659ac","Type":"ContainerStarted","Data":"1659b7a0a524ba1ff329df159f0670f6319fd06ca8116817c423389e19a951cc"} Jul 07 15:42:02.273229 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:42:02.273194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f79c85b7-t6gjn" event={"ID":"d8b67002-9055-48bb-a9e9-637b820659ac","Type":"ContainerStarted","Data":"5d58d4b970c83a457aa3eb2f41f289cd8ca2938bc563c753c3ffa2aa44b9c5a8"} Jul 07 15:42:02.294668 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:42:02.294619 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78f79c85b7-t6gjn" podStartSLOduration=2.225404478 podStartE2EDuration="5.294602505s" totalImagesPullingTime="3.069198027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:02.293545107 +0000 UTC m=+91.892614425" watchObservedRunningTime="2026-07-07 15:42:02.294602505 +0000 UTC m=+91.893671789" Jul 07 15:42:08.105613 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:42:08.105578 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:42:08.106122 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:42:08.105652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:42:08.110249 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:42:08.110225 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:42:08.297454 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:42:08.297427 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:42:14.198616 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:42:14.198590 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ltsg9" Jul 07 15:43:24.431067 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:24.430989 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78f79c85b7-t6gjn"] Jul 07 15:43:49.451518 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.451455 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-78f79c85b7-t6gjn" podUID="d8b67002-9055-48bb-a9e9-637b820659ac" containerName="console" containerID="cri-o://5d58d4b970c83a457aa3eb2f41f289cd8ca2938bc563c753c3ffa2aa44b9c5a8" gracePeriod=15 Jul 07 15:43:49.630901 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.630872 2576 generic.go:356] "Generic (PLEG): container finished" podID="d8b67002-9055-48bb-a9e9-637b820659ac" containerID="5d58d4b970c83a457aa3eb2f41f289cd8ca2938bc563c753c3ffa2aa44b9c5a8" exitCode=0 Jul 07 15:43:49.631080 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.630955 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f79c85b7-t6gjn" event={"ID":"d8b67002-9055-48bb-a9e9-637b820659ac","Type":"ContainerDied","Data":"5d58d4b970c83a457aa3eb2f41f289cd8ca2938bc563c753c3ffa2aa44b9c5a8"} Jul 07 15:43:49.693895 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.693873 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:43:49.804614 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.804522 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-oauth-serving-cert\") pod \"d8b67002-9055-48bb-a9e9-637b820659ac\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " Jul 07 15:43:49.804614 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.804558 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-trusted-ca-bundle\") pod \"d8b67002-9055-48bb-a9e9-637b820659ac\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " Jul 07 15:43:49.804614 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.804576 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-oauth-config\") pod \"d8b67002-9055-48bb-a9e9-637b820659ac\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " Jul 07 15:43:49.804860 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.804627 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-serving-cert\") pod \"d8b67002-9055-48bb-a9e9-637b820659ac\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " Jul 07 15:43:49.804860 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.804677 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwn7l\" (UniqueName: \"kubernetes.io/projected/d8b67002-9055-48bb-a9e9-637b820659ac-kube-api-access-mwn7l\") pod \"d8b67002-9055-48bb-a9e9-637b820659ac\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " Jul 07 15:43:49.804860 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.804701 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-console-config\") pod \"d8b67002-9055-48bb-a9e9-637b820659ac\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " Jul 07 15:43:49.804860 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.804720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-service-ca\") pod \"d8b67002-9055-48bb-a9e9-637b820659ac\" (UID: \"d8b67002-9055-48bb-a9e9-637b820659ac\") " Jul 07 15:43:49.805082 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.804959 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8b67002-9055-48bb-a9e9-637b820659ac" (UID: "d8b67002-9055-48bb-a9e9-637b820659ac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:49.805136 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.805087 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8b67002-9055-48bb-a9e9-637b820659ac" (UID: "d8b67002-9055-48bb-a9e9-637b820659ac"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:49.805239 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.805215 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8b67002-9055-48bb-a9e9-637b820659ac" (UID: "d8b67002-9055-48bb-a9e9-637b820659ac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:49.805239 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.805222 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-console-config" (OuterVolumeSpecName: "console-config") pod "d8b67002-9055-48bb-a9e9-637b820659ac" (UID: "d8b67002-9055-48bb-a9e9-637b820659ac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:43:49.807169 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.807144 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8b67002-9055-48bb-a9e9-637b820659ac" (UID: "d8b67002-9055-48bb-a9e9-637b820659ac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:49.807272 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.807179 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8b67002-9055-48bb-a9e9-637b820659ac" (UID: "d8b67002-9055-48bb-a9e9-637b820659ac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:43:49.807272 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.807191 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b67002-9055-48bb-a9e9-637b820659ac-kube-api-access-mwn7l" (OuterVolumeSpecName: "kube-api-access-mwn7l") pod "d8b67002-9055-48bb-a9e9-637b820659ac" (UID: "d8b67002-9055-48bb-a9e9-637b820659ac"). InnerVolumeSpecName "kube-api-access-mwn7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:43:49.906042 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.905993 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-serving-cert\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:43:49.906042 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.906035 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwn7l\" (UniqueName: \"kubernetes.io/projected/d8b67002-9055-48bb-a9e9-637b820659ac-kube-api-access-mwn7l\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:43:49.906042 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.906047 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-console-config\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:43:49.906042 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.906057 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-service-ca\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:43:49.906042 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.906065 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-oauth-serving-cert\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:43:49.906334 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.906073 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8b67002-9055-48bb-a9e9-637b820659ac-trusted-ca-bundle\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:43:49.906334 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:49.906081 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8b67002-9055-48bb-a9e9-637b820659ac-console-oauth-config\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:43:50.635097 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:50.635064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78f79c85b7-t6gjn" event={"ID":"d8b67002-9055-48bb-a9e9-637b820659ac","Type":"ContainerDied","Data":"1659b7a0a524ba1ff329df159f0670f6319fd06ca8116817c423389e19a951cc"} Jul 07 15:43:50.635468 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:50.635123 2576 scope.go:117] "RemoveContainer" containerID="5d58d4b970c83a457aa3eb2f41f289cd8ca2938bc563c753c3ffa2aa44b9c5a8" Jul 07 15:43:50.635468 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:50.635078 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78f79c85b7-t6gjn" Jul 07 15:43:50.657839 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:50.657813 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78f79c85b7-t6gjn"] Jul 07 15:43:50.662413 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:50.662392 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78f79c85b7-t6gjn"] Jul 07 15:43:50.805327 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:43:50.805295 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b67002-9055-48bb-a9e9-637b820659ac" path="/var/lib/kubelet/pods/d8b67002-9055-48bb-a9e9-637b820659ac/volumes" Jul 07 15:45:01.699347 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.699252 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9"] Jul 07 15:45:01.699804 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.699548 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b67002-9055-48bb-a9e9-637b820659ac" containerName="console" Jul 07 15:45:01.699804 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.699560 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b67002-9055-48bb-a9e9-637b820659ac" containerName="console" Jul 07 15:45:01.699804 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.699608 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b67002-9055-48bb-a9e9-637b820659ac" containerName="console" Jul 07 15:45:01.702479 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.702465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.708709 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.708639 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 15:45:01.708849 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.708767 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 15:45:01.708913 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.708894 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 15:45:01.709496 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.709477 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 15:45:01.709848 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.709831 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z46jf\"" Jul 07 15:45:01.744541 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.744508 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9"] Jul 07 15:45:01.790307 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.790280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/787274ec-e837-43e3-bf5d-d246be0b8711-manifests\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.790470 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.790328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/787274ec-e837-43e3-bf5d-d246be0b8711-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.790470 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.790378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mj94\" (UniqueName: \"kubernetes.io/projected/787274ec-e837-43e3-bf5d-d246be0b8711-kube-api-access-6mj94\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.790470 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.790408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/787274ec-e837-43e3-bf5d-d246be0b8711-webhook-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.790470 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.790434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/787274ec-e837-43e3-bf5d-d246be0b8711-tmp\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.891519 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.891488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/787274ec-e837-43e3-bf5d-d246be0b8711-tmp\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.891718 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.891537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/787274ec-e837-43e3-bf5d-d246be0b8711-manifests\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.891718 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.891570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/787274ec-e837-43e3-bf5d-d246be0b8711-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.891718 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.891586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mj94\" (UniqueName: \"kubernetes.io/projected/787274ec-e837-43e3-bf5d-d246be0b8711-kube-api-access-6mj94\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.891718 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.891603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/787274ec-e837-43e3-bf5d-d246be0b8711-webhook-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.892053 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.892016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/787274ec-e837-43e3-bf5d-d246be0b8711-manifests\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.893883 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.893854 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/787274ec-e837-43e3-bf5d-d246be0b8711-tmp\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.894149 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.894130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/787274ec-e837-43e3-bf5d-d246be0b8711-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.894217 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.894156 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/787274ec-e837-43e3-bf5d-d246be0b8711-webhook-cert\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:01.902373 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:01.902350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mj94\" (UniqueName: \"kubernetes.io/projected/787274ec-e837-43e3-bf5d-d246be0b8711-kube-api-access-6mj94\") pod \"opendatahub-operator-controller-manager-77db849c7c-76jc9\" (UID: \"787274ec-e837-43e3-bf5d-d246be0b8711\") " pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:02.011663 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:02.011572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:02.143296 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:02.143271 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9"] Jul 07 15:45:02.145960 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:45:02.145931 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787274ec_e837_43e3_bf5d_d246be0b8711.slice/crio-cabb7c01b4d3da148953cf8f78e9b6e1122ac346ae3a765f06c1fd215008cb71 WatchSource:0}: Error finding container cabb7c01b4d3da148953cf8f78e9b6e1122ac346ae3a765f06c1fd215008cb71: Status 404 returned error can't find the container with id cabb7c01b4d3da148953cf8f78e9b6e1122ac346ae3a765f06c1fd215008cb71 Jul 07 15:45:02.862393 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:02.862356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" event={"ID":"787274ec-e837-43e3-bf5d-d246be0b8711","Type":"ContainerStarted","Data":"cabb7c01b4d3da148953cf8f78e9b6e1122ac346ae3a765f06c1fd215008cb71"} Jul 07 15:45:05.875232 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:05.875197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" event={"ID":"787274ec-e837-43e3-bf5d-d246be0b8711","Type":"ContainerStarted","Data":"d8d5673d0379be19785b98d020f38a3825114a6fa8afb216b4fe4c8214350608"} Jul 07 15:45:06.880552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:06.880464 2576 generic.go:356] "Generic (PLEG): container finished" podID="787274ec-e837-43e3-bf5d-d246be0b8711" containerID="d8d5673d0379be19785b98d020f38a3825114a6fa8afb216b4fe4c8214350608" exitCode=0 Jul 07 15:45:06.880552 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:06.880516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" event={"ID":"787274ec-e837-43e3-bf5d-d246be0b8711","Type":"ContainerDied","Data":"d8d5673d0379be19785b98d020f38a3825114a6fa8afb216b4fe4c8214350608"} Jul 07 15:45:07.886923 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:07.886879 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" event={"ID":"787274ec-e837-43e3-bf5d-d246be0b8711","Type":"ContainerStarted","Data":"8d059c4f40ed57b4a5475e63013a4350bc0946af107425637ed5056830b380c0"} Jul 07 15:45:07.887410 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:07.887011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:07.912874 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:07.912813 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" podStartSLOduration=1.996941675 podStartE2EDuration="6.912791418s" totalImagesPullingTime="3.862171642s" totalInitContainerRuntime="1.053678101s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:07.911333302 +0000 UTC m=+277.510402606" watchObservedRunningTime="2026-07-07 15:45:07.912791418 +0000 UTC m=+277.511860704" Jul 07 15:45:18.893068 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:18.893039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77db849c7c-76jc9" Jul 07 15:45:22.135219 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.135181 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cpd72"] Jul 07 15:45:22.138874 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.138852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:22.142242 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.142222 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 15:45:22.142345 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.142282 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2rmzn\"" Jul 07 15:45:22.147957 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.147934 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cpd72"] Jul 07 15:45:22.245413 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.245379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc84z\" (UniqueName: \"kubernetes.io/projected/28547c6f-99be-4a11-9257-fda364906d05-kube-api-access-cc84z\") pod \"odh-model-controller-7998c67dc4-cpd72\" (UID: \"28547c6f-99be-4a11-9257-fda364906d05\") " pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:22.245579 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.245430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28547c6f-99be-4a11-9257-fda364906d05-cert\") pod \"odh-model-controller-7998c67dc4-cpd72\" (UID: \"28547c6f-99be-4a11-9257-fda364906d05\") " pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:22.346144 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.346099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28547c6f-99be-4a11-9257-fda364906d05-cert\") pod \"odh-model-controller-7998c67dc4-cpd72\" (UID: \"28547c6f-99be-4a11-9257-fda364906d05\") " pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:22.346339 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.346173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc84z\" (UniqueName: \"kubernetes.io/projected/28547c6f-99be-4a11-9257-fda364906d05-kube-api-access-cc84z\") pod \"odh-model-controller-7998c67dc4-cpd72\" (UID: \"28547c6f-99be-4a11-9257-fda364906d05\") " pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:22.348643 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.348617 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28547c6f-99be-4a11-9257-fda364906d05-cert\") pod \"odh-model-controller-7998c67dc4-cpd72\" (UID: \"28547c6f-99be-4a11-9257-fda364906d05\") " pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:22.354651 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.354621 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc84z\" (UniqueName: \"kubernetes.io/projected/28547c6f-99be-4a11-9257-fda364906d05-kube-api-access-cc84z\") pod \"odh-model-controller-7998c67dc4-cpd72\" (UID: \"28547c6f-99be-4a11-9257-fda364906d05\") " pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:22.449373 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.449279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:22.585666 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.585635 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cpd72"] Jul 07 15:45:22.589256 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:45:22.589225 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28547c6f_99be_4a11_9257_fda364906d05.slice/crio-f21865a84c56ef06749d2ac77bf0bf896ff7eceee319600200575017b8e8040c WatchSource:0}: Error finding container f21865a84c56ef06749d2ac77bf0bf896ff7eceee319600200575017b8e8040c: Status 404 returned error can't find the container with id f21865a84c56ef06749d2ac77bf0bf896ff7eceee319600200575017b8e8040c Jul 07 15:45:22.942358 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:22.942322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" event={"ID":"28547c6f-99be-4a11-9257-fda364906d05","Type":"ContainerStarted","Data":"f21865a84c56ef06749d2ac77bf0bf896ff7eceee319600200575017b8e8040c"} Jul 07 15:45:25.956909 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:25.956824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" event={"ID":"28547c6f-99be-4a11-9257-fda364906d05","Type":"ContainerStarted","Data":"a2def92b1f133a59bf3a586ae5a9963fb7488e6a650b6792b923cef9a120c75f"} Jul 07 15:45:25.957341 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:25.956984 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:25.977978 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:25.977908 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" podStartSLOduration=0.95778467 podStartE2EDuration="3.9778868s" totalImagesPullingTime="3.02010213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:25.976445314 +0000 UTC m=+295.575514617" watchObservedRunningTime="2026-07-07 15:45:25.9778868 +0000 UTC m=+295.576956084" Jul 07 15:45:27.317788 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.317755 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6669df4868-vm26x"] Jul 07 15:45:27.320429 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.320401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:27.323302 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.323281 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5c8jv\"" Jul 07 15:45:27.323403 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.323315 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 15:45:27.324315 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.324293 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 15:45:27.324402 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.324332 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 15:45:27.324575 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.324561 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 15:45:27.336221 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.336196 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6669df4868-vm26x"] Jul 07 15:45:27.495556 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.495520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a1f55ef-5235-42be-872b-483bc1f97143-tmp\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:27.495733 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.495592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lpq\" (UniqueName: \"kubernetes.io/projected/7a1f55ef-5235-42be-872b-483bc1f97143-kube-api-access-z4lpq\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:27.495733 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.495624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1f55ef-5235-42be-872b-483bc1f97143-tls-certs\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:27.596883 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.596785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a1f55ef-5235-42be-872b-483bc1f97143-tmp\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:27.596883 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.596844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lpq\" (UniqueName: \"kubernetes.io/projected/7a1f55ef-5235-42be-872b-483bc1f97143-kube-api-access-z4lpq\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:27.597113 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.596904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1f55ef-5235-42be-872b-483bc1f97143-tls-certs\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:27.597113 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:45:27.597052 2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 07 15:45:27.597199 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:45:27.597144 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a1f55ef-5235-42be-872b-483bc1f97143-tls-certs podName:7a1f55ef-5235-42be-872b-483bc1f97143 nodeName:}" failed. No retries permitted until 2026-07-07 15:45:28.097123077 +0000 UTC m=+297.696192340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7a1f55ef-5235-42be-872b-483bc1f97143-tls-certs") pod "kube-auth-proxy-6669df4868-vm26x" (UID: "7a1f55ef-5235-42be-872b-483bc1f97143") : secret "kube-auth-proxy-tls" not found Jul 07 15:45:27.599273 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.599257 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a1f55ef-5235-42be-872b-483bc1f97143-tmp\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:27.610092 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:27.610067 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lpq\" (UniqueName: \"kubernetes.io/projected/7a1f55ef-5235-42be-872b-483bc1f97143-kube-api-access-z4lpq\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:28.101948 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:28.101915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1f55ef-5235-42be-872b-483bc1f97143-tls-certs\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:28.104369 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:28.104347 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1f55ef-5235-42be-872b-483bc1f97143-tls-certs\") pod \"kube-auth-proxy-6669df4868-vm26x\" (UID: \"7a1f55ef-5235-42be-872b-483bc1f97143\") " pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:28.231765 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:28.231726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" Jul 07 15:45:28.386460 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:28.386431 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6669df4868-vm26x"] Jul 07 15:45:28.389134 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:45:28.389109 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1f55ef_5235_42be_872b_483bc1f97143.slice/crio-acd484946d42edfbd1259bfb549fc32d7430408dca65fe0c454d66c2ce7c1eec WatchSource:0}: Error finding container acd484946d42edfbd1259bfb549fc32d7430408dca65fe0c454d66c2ce7c1eec: Status 404 returned error can't find the container with id acd484946d42edfbd1259bfb549fc32d7430408dca65fe0c454d66c2ce7c1eec Jul 07 15:45:28.968876 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:28.968837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" event={"ID":"7a1f55ef-5235-42be-872b-483bc1f97143","Type":"ContainerStarted","Data":"acd484946d42edfbd1259bfb549fc32d7430408dca65fe0c454d66c2ce7c1eec"} Jul 07 15:45:29.414406 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.414364 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-778944b866-zlbn9"] Jul 07 15:45:29.417580 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.417557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.421528 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.421507 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:45:29.422834 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.422813 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 15:45:29.422962 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.422906 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 15:45:29.423115 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.423097 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 15:45:29.442098 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.442068 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 15:45:29.442331 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.442301 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mv4kw\"" Jul 07 15:45:29.443389 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.443364 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778944b866-zlbn9"] Jul 07 15:45:29.513290 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.513262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-cert\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.513486 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.513310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwgf\" (UniqueName: \"kubernetes.io/projected/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-kube-api-access-wzwgf\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.513486 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.513366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-metrics-cert\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.513486 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.513393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-manager-config\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.613961 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.613928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-cert\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.614170 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.614027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwgf\" (UniqueName: \"kubernetes.io/projected/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-kube-api-access-wzwgf\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.614170 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.614076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-metrics-cert\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.614170 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.614103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-manager-config\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.614721 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.614698 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-manager-config\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.616623 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.616601 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-metrics-cert\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.616705 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.616641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-cert\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.627875 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.627850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwgf\" (UniqueName: \"kubernetes.io/projected/fb4e4aa4-dca6-4e0e-b575-3f74432b8c31-kube-api-access-wzwgf\") pod \"lws-controller-manager-778944b866-zlbn9\" (UID: \"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31\") " pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.729118 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.729046 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:29.863460 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.863422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778944b866-zlbn9"] Jul 07 15:45:29.865442 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:45:29.865415 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4e4aa4_dca6_4e0e_b575_3f74432b8c31.slice/crio-5cfa72b36baf33fec34d4199f607c876e74d1848839eeeba5e290768ab29af01 WatchSource:0}: Error finding container 5cfa72b36baf33fec34d4199f607c876e74d1848839eeeba5e290768ab29af01: Status 404 returned error can't find the container with id 5cfa72b36baf33fec34d4199f607c876e74d1848839eeeba5e290768ab29af01 Jul 07 15:45:29.973689 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:29.973654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" event={"ID":"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31","Type":"ContainerStarted","Data":"5cfa72b36baf33fec34d4199f607c876e74d1848839eeeba5e290768ab29af01"} Jul 07 15:45:30.742881 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:30.742856 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 15:45:32.810851 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:32.810831 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 15:45:32.987285 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:32.987238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" event={"ID":"7a1f55ef-5235-42be-872b-483bc1f97143","Type":"ContainerStarted","Data":"d1ac642654a1bbbf7d33486d83a6f9729f547a9d6aca3ccda034d6d85ae8aceb"} Jul 07 15:45:32.988822 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:32.988794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" event={"ID":"fb4e4aa4-dca6-4e0e-b575-3f74432b8c31","Type":"ContainerStarted","Data":"4479807d1fdf85e367309cae98aae5e464bbd0bd91e5185734314053f2c0e38a"} Jul 07 15:45:32.988930 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:32.988854 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:45:33.084121 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:33.084071 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6669df4868-vm26x" podStartSLOduration=1.667919422 podStartE2EDuration="6.084055675s" totalImagesPullingTime="4.416136253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:33.083526727 +0000 UTC m=+302.682596030" watchObservedRunningTime="2026-07-07 15:45:33.084055675 +0000 UTC m=+302.683124959" Jul 07 15:45:33.152740 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:33.152684 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" podStartSLOduration=1.162172564 podStartE2EDuration="4.15266451s" totalImagesPullingTime="2.990491946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:33.150677562 +0000 UTC m=+302.749746866" watchObservedRunningTime="2026-07-07 15:45:33.15266451 +0000 UTC m=+302.751733794" Jul 07 15:45:36.962878 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:36.962850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-cpd72" Jul 07 15:45:43.995511 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:45:43.995484 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-778944b866-zlbn9" Jul 07 15:48:11.947162 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:11.947131 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:48:11.950104 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:11.950088 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:11.953187 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:11.953167 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gnpsb\"" Jul 07 15:48:11.953331 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:11.953317 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 15:48:11.954505 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:11.954488 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 15:48:11.954598 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:11.954511 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 15:48:11.960068 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:11.960045 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:48:11.975256 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:11.975232 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:48:12.014810 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.014780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmb94\" (UniqueName: \"kubernetes.io/projected/cfca23ef-78fc-4d21-80f4-17d477df477e-kube-api-access-tmb94\") pod \"limitador-limitador-54d6d988d5-p89nr\" (UID: \"cfca23ef-78fc-4d21-80f4-17d477df477e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:12.015028 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.014827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfca23ef-78fc-4d21-80f4-17d477df477e-config-file\") pod \"limitador-limitador-54d6d988d5-p89nr\" (UID: \"cfca23ef-78fc-4d21-80f4-17d477df477e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:12.116102 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.116047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmb94\" (UniqueName: \"kubernetes.io/projected/cfca23ef-78fc-4d21-80f4-17d477df477e-kube-api-access-tmb94\") pod \"limitador-limitador-54d6d988d5-p89nr\" (UID: \"cfca23ef-78fc-4d21-80f4-17d477df477e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:12.116102 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.116105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfca23ef-78fc-4d21-80f4-17d477df477e-config-file\") pod \"limitador-limitador-54d6d988d5-p89nr\" (UID: \"cfca23ef-78fc-4d21-80f4-17d477df477e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:12.116688 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.116671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfca23ef-78fc-4d21-80f4-17d477df477e-config-file\") pod \"limitador-limitador-54d6d988d5-p89nr\" (UID: \"cfca23ef-78fc-4d21-80f4-17d477df477e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:12.124511 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.124487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmb94\" (UniqueName: \"kubernetes.io/projected/cfca23ef-78fc-4d21-80f4-17d477df477e-kube-api-access-tmb94\") pod \"limitador-limitador-54d6d988d5-p89nr\" (UID: \"cfca23ef-78fc-4d21-80f4-17d477df477e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:12.261177 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.261099 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:12.395370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.395339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:48:12.398140 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:48:12.398101 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfca23ef_78fc_4d21_80f4_17d477df477e.slice/crio-5c899ecac292a6d29439a8c891ea174d9d08f9904b68e2603f16d15efe4ae452 WatchSource:0}: Error finding container 5c899ecac292a6d29439a8c891ea174d9d08f9904b68e2603f16d15efe4ae452: Status 404 returned error can't find the container with id 5c899ecac292a6d29439a8c891ea174d9d08f9904b68e2603f16d15efe4ae452 Jul 07 15:48:12.400120 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.400101 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:48:12.547954 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.547871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" event={"ID":"cfca23ef-78fc-4d21-80f4-17d477df477e","Type":"ContainerStarted","Data":"5c899ecac292a6d29439a8c891ea174d9d08f9904b68e2603f16d15efe4ae452"} Jul 07 15:48:12.641764 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.641735 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-lz8w7"] Jul 07 15:48:12.645869 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.645852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lz8w7" Jul 07 15:48:12.648449 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.648428 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rn7tr\"" Jul 07 15:48:12.653475 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.653455 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lz8w7"] Jul 07 15:48:12.720199 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.720166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpc6\" (UniqueName: \"kubernetes.io/projected/88ff89fd-bc0c-4910-b1af-f1992d499c6c-kube-api-access-9zpc6\") pod \"authorino-75d556bd77-lz8w7\" (UID: \"88ff89fd-bc0c-4910-b1af-f1992d499c6c\") " pod="kuadrant-system/authorino-75d556bd77-lz8w7" Jul 07 15:48:12.821301 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.821274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpc6\" (UniqueName: \"kubernetes.io/projected/88ff89fd-bc0c-4910-b1af-f1992d499c6c-kube-api-access-9zpc6\") pod \"authorino-75d556bd77-lz8w7\" (UID: \"88ff89fd-bc0c-4910-b1af-f1992d499c6c\") " pod="kuadrant-system/authorino-75d556bd77-lz8w7" Jul 07 15:48:12.830867 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.830836 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpc6\" (UniqueName: \"kubernetes.io/projected/88ff89fd-bc0c-4910-b1af-f1992d499c6c-kube-api-access-9zpc6\") pod \"authorino-75d556bd77-lz8w7\" (UID: \"88ff89fd-bc0c-4910-b1af-f1992d499c6c\") " pod="kuadrant-system/authorino-75d556bd77-lz8w7" Jul 07 15:48:12.955645 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:12.955611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lz8w7" Jul 07 15:48:13.132355 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:13.132223 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lz8w7"] Jul 07 15:48:13.141851 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:48:13.141819 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88ff89fd_bc0c_4910_b1af_f1992d499c6c.slice/crio-857cd0c48a3cf89e8ec75f3045b72a255f02e6682372b7dd785e3a08eba1a0b1 WatchSource:0}: Error finding container 857cd0c48a3cf89e8ec75f3045b72a255f02e6682372b7dd785e3a08eba1a0b1: Status 404 returned error can't find the container with id 857cd0c48a3cf89e8ec75f3045b72a255f02e6682372b7dd785e3a08eba1a0b1 Jul 07 15:48:13.559219 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:13.558412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lz8w7" event={"ID":"88ff89fd-bc0c-4910-b1af-f1992d499c6c","Type":"ContainerStarted","Data":"857cd0c48a3cf89e8ec75f3045b72a255f02e6682372b7dd785e3a08eba1a0b1"} Jul 07 15:48:17.576402 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:17.576360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" event={"ID":"cfca23ef-78fc-4d21-80f4-17d477df477e","Type":"ContainerStarted","Data":"71c217ebdc67bca3d0c25ebf97a1d0ba8a3a05527d679af6615e949928479adb"} Jul 07 15:48:17.576888 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:17.576476 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:17.578019 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:17.577994 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lz8w7" event={"ID":"88ff89fd-bc0c-4910-b1af-f1992d499c6c","Type":"ContainerStarted","Data":"7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9"} Jul 07 15:48:17.595174 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:17.595127 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" podStartSLOduration=2.088446635 podStartE2EDuration="6.595109931s" totalImagesPullingTime="4.506663296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:17.594843592 +0000 UTC m=+467.193912892" watchObservedRunningTime="2026-07-07 15:48:17.595109931 +0000 UTC m=+467.194179215" Jul 07 15:48:17.610893 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:17.610846 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-lz8w7" podStartSLOduration=1.841569937 podStartE2EDuration="5.610832588s" totalImagesPullingTime="3.769262651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:17.609817633 +0000 UTC m=+467.208886919" watchObservedRunningTime="2026-07-07 15:48:17.610832588 +0000 UTC m=+467.209901871" Jul 07 15:48:28.583128 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:28.583099 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-p89nr" Jul 07 15:48:46.343062 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.343026 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-29ndh"] Jul 07 15:48:46.352526 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.352499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-29ndh" Jul 07 15:48:46.353182 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.353155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-29ndh"] Jul 07 15:48:46.379935 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.379910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phx5\" (UniqueName: \"kubernetes.io/projected/c40567d4-3093-4897-ab02-e3ae6e20231b-kube-api-access-4phx5\") pod \"authorino-5d4b67c446-29ndh\" (UID: \"c40567d4-3093-4897-ab02-e3ae6e20231b\") " pod="kuadrant-system/authorino-5d4b67c446-29ndh" Jul 07 15:48:46.480666 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.480629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phx5\" (UniqueName: \"kubernetes.io/projected/c40567d4-3093-4897-ab02-e3ae6e20231b-kube-api-access-4phx5\") pod \"authorino-5d4b67c446-29ndh\" (UID: \"c40567d4-3093-4897-ab02-e3ae6e20231b\") " pod="kuadrant-system/authorino-5d4b67c446-29ndh" Jul 07 15:48:46.489050 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.489021 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phx5\" (UniqueName: \"kubernetes.io/projected/c40567d4-3093-4897-ab02-e3ae6e20231b-kube-api-access-4phx5\") pod \"authorino-5d4b67c446-29ndh\" (UID: \"c40567d4-3093-4897-ab02-e3ae6e20231b\") " pod="kuadrant-system/authorino-5d4b67c446-29ndh" Jul 07 15:48:46.578978 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.578949 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-29ndh"] Jul 07 15:48:46.579222 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.579210 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-29ndh" Jul 07 15:48:46.604333 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.604241 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5789565f95-pn7wr"] Jul 07 15:48:46.607738 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.607714 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5789565f95-pn7wr" Jul 07 15:48:46.616028 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.616002 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5789565f95-pn7wr"] Jul 07 15:48:46.683250 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.682936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnngg\" (UniqueName: \"kubernetes.io/projected/3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f-kube-api-access-fnngg\") pod \"authorino-5789565f95-pn7wr\" (UID: \"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f\") " pod="kuadrant-system/authorino-5789565f95-pn7wr" Jul 07 15:48:46.711494 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.711465 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-29ndh"] Jul 07 15:48:46.714174 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:48:46.714143 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc40567d4_3093_4897_ab02_e3ae6e20231b.slice/crio-a6bae7dde987a9fcde58602602cf19dfa066e47a085e45e02d655b4975809d52 WatchSource:0}: Error finding container a6bae7dde987a9fcde58602602cf19dfa066e47a085e45e02d655b4975809d52: Status 404 returned error can't find the container with id a6bae7dde987a9fcde58602602cf19dfa066e47a085e45e02d655b4975809d52 Jul 07 15:48:46.783400 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.783362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnngg\" (UniqueName: \"kubernetes.io/projected/3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f-kube-api-access-fnngg\") pod \"authorino-5789565f95-pn7wr\" (UID: \"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f\") " pod="kuadrant-system/authorino-5789565f95-pn7wr" Jul 07 15:48:46.792781 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.792745 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnngg\" (UniqueName: \"kubernetes.io/projected/3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f-kube-api-access-fnngg\") pod \"authorino-5789565f95-pn7wr\" (UID: \"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f\") " pod="kuadrant-system/authorino-5789565f95-pn7wr" Jul 07 15:48:46.871591 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.871502 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5789565f95-pn7wr"] Jul 07 15:48:46.871752 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.871729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5789565f95-pn7wr" Jul 07 15:48:46.898424 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.898392 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d49f58667-ct948"] Jul 07 15:48:46.901887 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.901863 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:48:46.904671 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.904647 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 15:48:46.910348 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.910322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d49f58667-ct948"] Jul 07 15:48:46.984734 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.984699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41b83d67-29c9-4169-b70a-01cdb2264a19-tls-cert\") pod \"authorino-5d49f58667-ct948\" (UID: \"41b83d67-29c9-4169-b70a-01cdb2264a19\") " pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:48:46.984890 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:46.984761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m746v\" (UniqueName: \"kubernetes.io/projected/41b83d67-29c9-4169-b70a-01cdb2264a19-kube-api-access-m746v\") pod \"authorino-5d49f58667-ct948\" (UID: \"41b83d67-29c9-4169-b70a-01cdb2264a19\") " pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:48:47.003041 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.003001 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5789565f95-pn7wr"] Jul 07 15:48:47.005754 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:48:47.005720 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a281c76_b9a7_46b9_9b5a_d6b1ad602c2f.slice/crio-99a79747e862ccc2c3e451f7d5242525011c0b97a55119e7c1bddf5627aa94aa WatchSource:0}: Error finding container 99a79747e862ccc2c3e451f7d5242525011c0b97a55119e7c1bddf5627aa94aa: Status 404 returned error can't find the container with id 99a79747e862ccc2c3e451f7d5242525011c0b97a55119e7c1bddf5627aa94aa Jul 07 15:48:47.085231 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.085207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m746v\" (UniqueName: \"kubernetes.io/projected/41b83d67-29c9-4169-b70a-01cdb2264a19-kube-api-access-m746v\") pod \"authorino-5d49f58667-ct948\" (UID: \"41b83d67-29c9-4169-b70a-01cdb2264a19\") " pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:48:47.085358 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.085275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41b83d67-29c9-4169-b70a-01cdb2264a19-tls-cert\") pod \"authorino-5d49f58667-ct948\" (UID: \"41b83d67-29c9-4169-b70a-01cdb2264a19\") " pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:48:47.087777 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.087760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41b83d67-29c9-4169-b70a-01cdb2264a19-tls-cert\") pod \"authorino-5d49f58667-ct948\" (UID: \"41b83d67-29c9-4169-b70a-01cdb2264a19\") " pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:48:47.093468 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.093449 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m746v\" (UniqueName: \"kubernetes.io/projected/41b83d67-29c9-4169-b70a-01cdb2264a19-kube-api-access-m746v\") pod \"authorino-5d49f58667-ct948\" (UID: \"41b83d67-29c9-4169-b70a-01cdb2264a19\") " pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:48:47.212204 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.212171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:48:47.343272 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.343245 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d49f58667-ct948"] Jul 07 15:48:47.348033 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:48:47.348010 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41b83d67_29c9_4169_b70a_01cdb2264a19.slice/crio-3eba427890b2a126a00168f17d3812ff4969e8be43206844f733af1ca565bc92 WatchSource:0}: Error finding container 3eba427890b2a126a00168f17d3812ff4969e8be43206844f733af1ca565bc92: Status 404 returned error can't find the container with id 3eba427890b2a126a00168f17d3812ff4969e8be43206844f733af1ca565bc92 Jul 07 15:48:47.687157 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.687116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-29ndh" event={"ID":"c40567d4-3093-4897-ab02-e3ae6e20231b","Type":"ContainerStarted","Data":"eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44"} Jul 07 15:48:47.687157 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.687137 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-29ndh" podUID="c40567d4-3093-4897-ab02-e3ae6e20231b" containerName="authorino" containerID="cri-o://eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44" gracePeriod=30 Jul 07 15:48:47.687157 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.687167 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-29ndh" event={"ID":"c40567d4-3093-4897-ab02-e3ae6e20231b","Type":"ContainerStarted","Data":"a6bae7dde987a9fcde58602602cf19dfa066e47a085e45e02d655b4975809d52"} Jul 07 15:48:47.688475 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.688444 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d49f58667-ct948" event={"ID":"41b83d67-29c9-4169-b70a-01cdb2264a19","Type":"ContainerStarted","Data":"3eba427890b2a126a00168f17d3812ff4969e8be43206844f733af1ca565bc92"} Jul 07 15:48:47.689922 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.689896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5789565f95-pn7wr" event={"ID":"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f","Type":"ContainerStarted","Data":"c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549"} Jul 07 15:48:47.690041 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.689929 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5789565f95-pn7wr" event={"ID":"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f","Type":"ContainerStarted","Data":"99a79747e862ccc2c3e451f7d5242525011c0b97a55119e7c1bddf5627aa94aa"} Jul 07 15:48:47.690041 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.689985 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5789565f95-pn7wr" podUID="3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f" containerName="authorino" containerID="cri-o://c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549" gracePeriod=30 Jul 07 15:48:47.703791 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.703743 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-29ndh" podStartSLOduration=1.337365622 podStartE2EDuration="1.703728008s" totalImagesPullingTime="366.362386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:47.702807184 +0000 UTC m=+497.301876466" watchObservedRunningTime="2026-07-07 15:48:47.703728008 +0000 UTC m=+497.302797290" Jul 07 15:48:47.718611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.718570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5789565f95-pn7wr" podStartSLOduration=1.403397091 podStartE2EDuration="1.71855433s" totalImagesPullingTime="315.157239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:47.717867711 +0000 UTC m=+497.316936996" watchObservedRunningTime="2026-07-07 15:48:47.71855433 +0000 UTC m=+497.317623625" Jul 07 15:48:47.954512 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.954489 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-29ndh" Jul 07 15:48:47.960561 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:47.960541 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5789565f95-pn7wr" Jul 07 15:48:48.090581 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.090542 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnngg\" (UniqueName: \"kubernetes.io/projected/3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f-kube-api-access-fnngg\") pod \"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f\" (UID: \"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f\") " Jul 07 15:48:48.090581 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.090585 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4phx5\" (UniqueName: \"kubernetes.io/projected/c40567d4-3093-4897-ab02-e3ae6e20231b-kube-api-access-4phx5\") pod \"c40567d4-3093-4897-ab02-e3ae6e20231b\" (UID: \"c40567d4-3093-4897-ab02-e3ae6e20231b\") " Jul 07 15:48:48.093040 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.093003 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f-kube-api-access-fnngg" (OuterVolumeSpecName: "kube-api-access-fnngg") pod "3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f" (UID: "3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f"). InnerVolumeSpecName "kube-api-access-fnngg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:48:48.093040 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.093003 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40567d4-3093-4897-ab02-e3ae6e20231b-kube-api-access-4phx5" (OuterVolumeSpecName: "kube-api-access-4phx5") pod "c40567d4-3093-4897-ab02-e3ae6e20231b" (UID: "c40567d4-3093-4897-ab02-e3ae6e20231b"). InnerVolumeSpecName "kube-api-access-4phx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:48:48.191646 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.191610 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnngg\" (UniqueName: \"kubernetes.io/projected/3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f-kube-api-access-fnngg\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:48:48.191646 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.191639 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4phx5\" (UniqueName: \"kubernetes.io/projected/c40567d4-3093-4897-ab02-e3ae6e20231b-kube-api-access-4phx5\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:48:48.695620 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.695573 2576 generic.go:356] "Generic (PLEG): container finished" podID="c40567d4-3093-4897-ab02-e3ae6e20231b" containerID="eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44" exitCode=0 Jul 07 15:48:48.695620 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.695624 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-29ndh" Jul 07 15:48:48.696123 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.695660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-29ndh" event={"ID":"c40567d4-3093-4897-ab02-e3ae6e20231b","Type":"ContainerDied","Data":"eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44"} Jul 07 15:48:48.696123 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.695702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-29ndh" event={"ID":"c40567d4-3093-4897-ab02-e3ae6e20231b","Type":"ContainerDied","Data":"a6bae7dde987a9fcde58602602cf19dfa066e47a085e45e02d655b4975809d52"} Jul 07 15:48:48.696123 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.695723 2576 scope.go:117] "RemoveContainer" containerID="eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44" Jul 07 15:48:48.697235 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.697212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d49f58667-ct948" event={"ID":"41b83d67-29c9-4169-b70a-01cdb2264a19","Type":"ContainerStarted","Data":"add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b"} Jul 07 15:48:48.698546 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.698521 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f" containerID="c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549" exitCode=0 Jul 07 15:48:48.698651 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.698553 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5789565f95-pn7wr" Jul 07 15:48:48.698651 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.698566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5789565f95-pn7wr" event={"ID":"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f","Type":"ContainerDied","Data":"c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549"} Jul 07 15:48:48.698651 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.698593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5789565f95-pn7wr" event={"ID":"3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f","Type":"ContainerDied","Data":"99a79747e862ccc2c3e451f7d5242525011c0b97a55119e7c1bddf5627aa94aa"} Jul 07 15:48:48.704692 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.704447 2576 scope.go:117] "RemoveContainer" containerID="eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44" Jul 07 15:48:48.704857 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:48:48.704832 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44\": container with ID starting with eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44 not found: ID does not exist" containerID="eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44" Jul 07 15:48:48.704933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.704870 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44"} err="failed to get container status \"eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44\": rpc error: code = NotFound desc = could not find container \"eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44\": container with ID starting with eb3c8d1f84ab10f5d003db92d3b41ba3e847320f622efa2e7d7cd02f51a5be44 not found: ID does not exist" Jul 07 15:48:48.704933 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.704905 2576 scope.go:117] "RemoveContainer" containerID="c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549" Jul 07 15:48:48.712525 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.712509 2576 scope.go:117] "RemoveContainer" containerID="c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549" Jul 07 15:48:48.712782 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:48:48.712765 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549\": container with ID starting with c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549 not found: ID does not exist" containerID="c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549" Jul 07 15:48:48.712823 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.712791 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549"} err="failed to get container status \"c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549\": rpc error: code = NotFound desc = could not find container \"c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549\": container with ID starting with c9141390f47976feaed9efda101793c956f733a1d6258f716b6dab582d41c549 not found: ID does not exist" Jul 07 15:48:48.721069 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.721030 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d49f58667-ct948" podStartSLOduration=2.245241364 podStartE2EDuration="2.721015585s" totalImagesPullingTime="475.774221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:48.719218943 +0000 UTC m=+498.318288228" watchObservedRunningTime="2026-07-07 15:48:48.721015585 +0000 UTC m=+498.320084903" Jul 07 15:48:48.734960 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.734930 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-29ndh"] Jul 07 15:48:48.739018 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.738993 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-29ndh"] Jul 07 15:48:48.745038 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.744939 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lz8w7"] Jul 07 15:48:48.745191 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.745165 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-lz8w7" podUID="88ff89fd-bc0c-4910-b1af-f1992d499c6c" containerName="authorino" containerID="cri-o://7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9" gracePeriod=30 Jul 07 15:48:48.757399 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.757373 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5789565f95-pn7wr"] Jul 07 15:48:48.763080 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.763058 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5789565f95-pn7wr"] Jul 07 15:48:48.805530 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.805500 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f" path="/var/lib/kubelet/pods/3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f/volumes" Jul 07 15:48:48.805793 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.805780 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40567d4-3093-4897-ab02-e3ae6e20231b" path="/var/lib/kubelet/pods/c40567d4-3093-4897-ab02-e3ae6e20231b/volumes" Jul 07 15:48:48.987061 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:48.987035 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lz8w7" Jul 07 15:48:49.098374 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.098342 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zpc6\" (UniqueName: \"kubernetes.io/projected/88ff89fd-bc0c-4910-b1af-f1992d499c6c-kube-api-access-9zpc6\") pod \"88ff89fd-bc0c-4910-b1af-f1992d499c6c\" (UID: \"88ff89fd-bc0c-4910-b1af-f1992d499c6c\") " Jul 07 15:48:49.100517 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.100484 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ff89fd-bc0c-4910-b1af-f1992d499c6c-kube-api-access-9zpc6" (OuterVolumeSpecName: "kube-api-access-9zpc6") pod "88ff89fd-bc0c-4910-b1af-f1992d499c6c" (UID: "88ff89fd-bc0c-4910-b1af-f1992d499c6c"). InnerVolumeSpecName "kube-api-access-9zpc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:48:49.198843 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.198814 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zpc6\" (UniqueName: \"kubernetes.io/projected/88ff89fd-bc0c-4910-b1af-f1992d499c6c-kube-api-access-9zpc6\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:48:49.703543 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.703508 2576 generic.go:356] "Generic (PLEG): container finished" podID="88ff89fd-bc0c-4910-b1af-f1992d499c6c" containerID="7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9" exitCode=0 Jul 07 15:48:49.704027 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.703564 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lz8w7" Jul 07 15:48:49.704027 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.703593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lz8w7" event={"ID":"88ff89fd-bc0c-4910-b1af-f1992d499c6c","Type":"ContainerDied","Data":"7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9"} Jul 07 15:48:49.704027 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.703645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lz8w7" event={"ID":"88ff89fd-bc0c-4910-b1af-f1992d499c6c","Type":"ContainerDied","Data":"857cd0c48a3cf89e8ec75f3045b72a255f02e6682372b7dd785e3a08eba1a0b1"} Jul 07 15:48:49.704027 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.703663 2576 scope.go:117] "RemoveContainer" containerID="7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9" Jul 07 15:48:49.711889 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.711868 2576 scope.go:117] "RemoveContainer" containerID="7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9" Jul 07 15:48:49.712143 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:48:49.712126 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9\": container with ID starting with 7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9 not found: ID does not exist" containerID="7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9" Jul 07 15:48:49.712211 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.712152 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9"} err="failed to get container status \"7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9\": rpc error: code = NotFound desc = could not find container \"7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9\": container with ID starting with 7c537cbd38f4007733fbbcced66eb6e51cfaa01cf783470d5b398b02fe2765a9 not found: ID does not exist" Jul 07 15:48:49.724259 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.724233 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lz8w7"] Jul 07 15:48:49.726058 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:49.726038 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lz8w7"] Jul 07 15:48:50.806218 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:48:50.806189 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ff89fd-bc0c-4910-b1af-f1992d499c6c" path="/var/lib/kubelet/pods/88ff89fd-bc0c-4910-b1af-f1992d499c6c/volumes" Jul 07 15:49:30.248778 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:49:30.248686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:49:46.135282 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:49:46.135247 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:50:04.837149 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:04.837111 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:50:14.837027 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:14.836989 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:50:28.149914 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:28.149882 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:50:41.244368 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:41.244332 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:50:47.732255 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732225 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs"] Jul 07 15:50:47.732615 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732497 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ff89fd-bc0c-4910-b1af-f1992d499c6c" containerName="authorino" Jul 07 15:50:47.732615 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732534 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ff89fd-bc0c-4910-b1af-f1992d499c6c" containerName="authorino" Jul 07 15:50:47.732615 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732550 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40567d4-3093-4897-ab02-e3ae6e20231b" containerName="authorino" Jul 07 15:50:47.732615 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732557 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40567d4-3093-4897-ab02-e3ae6e20231b" containerName="authorino" Jul 07 15:50:47.732615 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732566 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f" containerName="authorino" Jul 07 15:50:47.732615 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732571 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f" containerName="authorino" Jul 07 15:50:47.732615 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732617 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ff89fd-bc0c-4910-b1af-f1992d499c6c" containerName="authorino" Jul 07 15:50:47.732822 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732625 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a281c76-b9a7-46b9-9b5a-d6b1ad602c2f" containerName="authorino" Jul 07 15:50:47.732822 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.732631 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40567d4-3093-4897-ab02-e3ae6e20231b" containerName="authorino" Jul 07 15:50:47.735370 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.735354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.738122 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.738102 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 15:50:47.738122 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.738102 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 15:50:47.739244 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.739228 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 15:50:47.739336 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.739300 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jn8gz\"" Jul 07 15:50:47.744403 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.744385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs"] Jul 07 15:50:47.777508 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.777477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djtb\" (UniqueName: \"kubernetes.io/projected/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-kube-api-access-7djtb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.777650 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.777519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.777650 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.777543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.777650 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.777635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.777756 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.777666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.777756 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.777701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.878374 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.878325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.878374 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.878383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.878611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.878444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.878611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.878485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djtb\" (UniqueName: \"kubernetes.io/projected/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-kube-api-access-7djtb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.878611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.878541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.878611 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.878577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.878830 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.878783 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.878889 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.878842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.879075 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.879049 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.880848 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.880827 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.881132 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.881115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:47.887601 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:47.887580 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djtb\" (UniqueName: \"kubernetes.io/projected/5e7616e2-b9a8-4858-b83c-4c99e24b3b72-kube-api-access-7djtb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs\" (UID: \"5e7616e2-b9a8-4858-b83c-4c99e24b3b72\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:48.044994 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:48.044867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:48.178961 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:48.178937 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs"] Jul 07 15:50:48.181467 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:50:48.181433 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e7616e2_b9a8_4858_b83c_4c99e24b3b72.slice/crio-047c216042d0dab7648d7a8041357a124d0e88d174b2958a9e9927bf0233669f WatchSource:0}: Error finding container 047c216042d0dab7648d7a8041357a124d0e88d174b2958a9e9927bf0233669f: Status 404 returned error can't find the container with id 047c216042d0dab7648d7a8041357a124d0e88d174b2958a9e9927bf0233669f Jul 07 15:50:49.156614 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:49.156571 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" event={"ID":"5e7616e2-b9a8-4858-b83c-4c99e24b3b72","Type":"ContainerStarted","Data":"047c216042d0dab7648d7a8041357a124d0e88d174b2958a9e9927bf0233669f"} Jul 07 15:50:50.136056 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:50.136022 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:50:50.161426 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:50.161391 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" event={"ID":"5e7616e2-b9a8-4858-b83c-4c99e24b3b72","Type":"ContainerStarted","Data":"5fb021b311c5048afe295b531be5e0650e37b7389563157ea829e4a502a81d65"} Jul 07 15:50:50.181771 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:50.181716 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" podStartSLOduration=2.121512433 podStartE2EDuration="3.181700554s" totalImagesPullingTime="1.060188121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:50.180701849 +0000 UTC m=+619.779771133" watchObservedRunningTime="2026-07-07 15:50:50.181700554 +0000 UTC m=+619.780769867" Jul 07 15:50:58.045694 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:58.045591 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:58.045694 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:58.045679 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:58.058536 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:58.058514 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:50:58.200919 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:50:58.200891 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs" Jul 07 15:51:50.168046 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.167997 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-798669dbf4-25m9v"] Jul 07 15:51:50.171296 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.171273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798669dbf4-25m9v" Jul 07 15:51:50.181451 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.181425 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-798669dbf4-25m9v"] Jul 07 15:51:50.282405 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.282350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqms\" (UniqueName: \"kubernetes.io/projected/0a2f9f6b-a0ff-44f7-89ad-b441384f1688-kube-api-access-zzqms\") pod \"authorino-798669dbf4-25m9v\" (UID: \"0a2f9f6b-a0ff-44f7-89ad-b441384f1688\") " pod="kuadrant-system/authorino-798669dbf4-25m9v" Jul 07 15:51:50.282593 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.282474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a2f9f6b-a0ff-44f7-89ad-b441384f1688-tls-cert\") pod \"authorino-798669dbf4-25m9v\" (UID: \"0a2f9f6b-a0ff-44f7-89ad-b441384f1688\") " pod="kuadrant-system/authorino-798669dbf4-25m9v" Jul 07 15:51:50.383804 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.383769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqms\" (UniqueName: \"kubernetes.io/projected/0a2f9f6b-a0ff-44f7-89ad-b441384f1688-kube-api-access-zzqms\") pod \"authorino-798669dbf4-25m9v\" (UID: \"0a2f9f6b-a0ff-44f7-89ad-b441384f1688\") " pod="kuadrant-system/authorino-798669dbf4-25m9v" Jul 07 15:51:50.384017 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.383859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a2f9f6b-a0ff-44f7-89ad-b441384f1688-tls-cert\") pod \"authorino-798669dbf4-25m9v\" (UID: \"0a2f9f6b-a0ff-44f7-89ad-b441384f1688\") " pod="kuadrant-system/authorino-798669dbf4-25m9v" Jul 07 15:51:50.386419 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.386391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a2f9f6b-a0ff-44f7-89ad-b441384f1688-tls-cert\") pod \"authorino-798669dbf4-25m9v\" (UID: \"0a2f9f6b-a0ff-44f7-89ad-b441384f1688\") " pod="kuadrant-system/authorino-798669dbf4-25m9v" Jul 07 15:51:50.392038 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.392008 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqms\" (UniqueName: \"kubernetes.io/projected/0a2f9f6b-a0ff-44f7-89ad-b441384f1688-kube-api-access-zzqms\") pod \"authorino-798669dbf4-25m9v\" (UID: \"0a2f9f6b-a0ff-44f7-89ad-b441384f1688\") " pod="kuadrant-system/authorino-798669dbf4-25m9v" Jul 07 15:51:50.479895 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.479804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798669dbf4-25m9v" Jul 07 15:51:50.619466 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:50.619439 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-798669dbf4-25m9v"] Jul 07 15:51:50.621957 ip-10-0-137-175 kubenswrapper[2576]: W0707 15:51:50.621932 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a2f9f6b_a0ff_44f7_89ad_b441384f1688.slice/crio-da23bfbd627381e7e87870de7ba8b58fc8b49220116ba3aeae781740ecae8923 WatchSource:0}: Error finding container da23bfbd627381e7e87870de7ba8b58fc8b49220116ba3aeae781740ecae8923: Status 404 returned error can't find the container with id da23bfbd627381e7e87870de7ba8b58fc8b49220116ba3aeae781740ecae8923 Jul 07 15:51:51.383440 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.383399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798669dbf4-25m9v" event={"ID":"0a2f9f6b-a0ff-44f7-89ad-b441384f1688","Type":"ContainerStarted","Data":"f0a9c1823f636530fd5d471ebf21a4001ce23158747f386aa25f77dcda40fe4f"} Jul 07 15:51:51.383440 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.383445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798669dbf4-25m9v" event={"ID":"0a2f9f6b-a0ff-44f7-89ad-b441384f1688","Type":"ContainerStarted","Data":"da23bfbd627381e7e87870de7ba8b58fc8b49220116ba3aeae781740ecae8923"} Jul 07 15:51:51.405634 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.405572 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-798669dbf4-25m9v" podStartSLOduration=0.992841841 podStartE2EDuration="1.405550841s" totalImagesPullingTime="412.709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:51.403505152 +0000 UTC m=+681.002574438" watchObservedRunningTime="2026-07-07 15:51:51.405550841 +0000 UTC m=+681.004620129" Jul 07 15:51:51.432910 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.432874 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d49f58667-ct948"] Jul 07 15:51:51.433114 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.433083 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d49f58667-ct948" podUID="41b83d67-29c9-4169-b70a-01cdb2264a19" containerName="authorino" containerID="cri-o://add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b" gracePeriod=30 Jul 07 15:51:51.690040 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.690009 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:51:51.796670 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.796626 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m746v\" (UniqueName: \"kubernetes.io/projected/41b83d67-29c9-4169-b70a-01cdb2264a19-kube-api-access-m746v\") pod \"41b83d67-29c9-4169-b70a-01cdb2264a19\" (UID: \"41b83d67-29c9-4169-b70a-01cdb2264a19\") " Jul 07 15:51:51.796846 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.796708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41b83d67-29c9-4169-b70a-01cdb2264a19-tls-cert\") pod \"41b83d67-29c9-4169-b70a-01cdb2264a19\" (UID: \"41b83d67-29c9-4169-b70a-01cdb2264a19\") " Jul 07 15:51:51.798904 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.798873 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b83d67-29c9-4169-b70a-01cdb2264a19-kube-api-access-m746v" (OuterVolumeSpecName: "kube-api-access-m746v") pod "41b83d67-29c9-4169-b70a-01cdb2264a19" (UID: "41b83d67-29c9-4169-b70a-01cdb2264a19"). InnerVolumeSpecName "kube-api-access-m746v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:51:51.808167 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.808137 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b83d67-29c9-4169-b70a-01cdb2264a19-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "41b83d67-29c9-4169-b70a-01cdb2264a19" (UID: "41b83d67-29c9-4169-b70a-01cdb2264a19"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:51:51.898226 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.898159 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m746v\" (UniqueName: \"kubernetes.io/projected/41b83d67-29c9-4169-b70a-01cdb2264a19-kube-api-access-m746v\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:51:51.898226 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:51.898196 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41b83d67-29c9-4169-b70a-01cdb2264a19-tls-cert\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 15:51:52.388514 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.388479 2576 generic.go:356] "Generic (PLEG): container finished" podID="41b83d67-29c9-4169-b70a-01cdb2264a19" containerID="add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b" exitCode=0 Jul 07 15:51:52.388938 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.388518 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d49f58667-ct948" Jul 07 15:51:52.388938 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.388575 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d49f58667-ct948" event={"ID":"41b83d67-29c9-4169-b70a-01cdb2264a19","Type":"ContainerDied","Data":"add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b"} Jul 07 15:51:52.388938 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.388626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d49f58667-ct948" event={"ID":"41b83d67-29c9-4169-b70a-01cdb2264a19","Type":"ContainerDied","Data":"3eba427890b2a126a00168f17d3812ff4969e8be43206844f733af1ca565bc92"} Jul 07 15:51:52.388938 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.388653 2576 scope.go:117] "RemoveContainer" containerID="add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b" Jul 07 15:51:52.397525 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.397510 2576 scope.go:117] "RemoveContainer" containerID="add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b" Jul 07 15:51:52.397829 ip-10-0-137-175 kubenswrapper[2576]: E0707 15:51:52.397813 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b\": container with ID starting with add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b not found: ID does not exist" containerID="add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b" Jul 07 15:51:52.397868 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.397838 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b"} err="failed to get container status \"add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b\": rpc error: code = NotFound desc = could not find container \"add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b\": container with ID starting with add90e58bb10de1d965967fa98631b3603aef1b28347e9d9fcc410083abbb27b not found: ID does not exist" Jul 07 15:51:52.412030 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.411960 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d49f58667-ct948"] Jul 07 15:51:52.415306 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.415280 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d49f58667-ct948"] Jul 07 15:51:52.806723 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:51:52.806632 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41b83d67-29c9-4169-b70a-01cdb2264a19" path="/var/lib/kubelet/pods/41b83d67-29c9-4169-b70a-01cdb2264a19/volumes" Jul 07 15:52:21.448502 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:52:21.448467 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:52:31.843129 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:52:31.843091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:52:41.539499 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:52:41.539462 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:52:51.757825 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:52:51.757792 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:53:02.246146 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:53:02.246112 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:53:30.935390 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:53:30.935358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:53:42.739516 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:53:42.739479 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:54:15.734838 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:54:15.734800 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:54:26.641400 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:54:26.641357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:54:35.035741 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:54:35.035706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:54:44.041557 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:54:44.041519 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:54:55.243739 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:54:55.243707 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:55:04.437433 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:55:04.437400 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:55:15.938285 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:55:15.938250 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:55:25.349766 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:55:25.349683 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:55:48.549916 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:55:48.549882 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:55:57.141929 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:55:57.141897 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:56:22.343057 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:56:22.343021 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:56:34.444685 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:56:34.444647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:56:48.160646 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:56:48.160606 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:56:58.373561 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:56:58.373478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:57:09.647729 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:57:09.647698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:57:18.553991 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:57:18.553935 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:57:27.944274 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:57:27.944242 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:59:17.846335 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:59:17.846298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:59:29.045528 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:59:29.045490 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:59:37.959319 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:59:37.959282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:59:48.248647 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:59:48.248615 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 15:59:59.552326 ip-10-0-137-175 kubenswrapper[2576]: I0707 15:59:59.552247 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:00:11.438413 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:00:11.438369 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:00:19.149220 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:00:19.149182 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:00:32.148387 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:00:32.148350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:00:41.342182 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:00:41.341558 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:00:49.943672 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:00:49.943637 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:01:02.739311 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:01:02.739268 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:01:20.945022 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:01:20.944983 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:01:29.846071 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:01:29.845985 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:01:38.638768 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:01:38.638733 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:01:50.838587 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:01:50.838552 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:02:05.841267 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:02:05.841233 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:02:16.634580 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:02:16.634543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:02:25.437357 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:02:25.437319 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:02:35.144823 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:02:35.144789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:02:44.627007 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:02:44.626956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:02:53.828708 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:02:53.828628 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:03:08.035360 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:03:08.035322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:03:17.052591 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:03:17.052549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:03:26.832251 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:03:26.832212 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:03:38.241348 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:03:38.241309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:03:50.443863 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:03:50.443825 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:03:59.066868 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:03:59.066832 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:04:10.589298 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:04:10.589249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:04:22.352500 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:04:22.352416 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:04:33.635598 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:04:33.635557 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:04:45.041044 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:04:45.041004 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:04:53.835070 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:04:53.835038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:05:03.139130 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:05:03.139093 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:05:21.031903 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:05:21.031864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:05:32.137687 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:05:32.137649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:07:10.142927 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:07:10.142893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:07:17.547092 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:07:17.547058 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:07:26.130553 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:07:26.130470 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:07:35.839164 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:07:35.839128 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:07:42.242092 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:07:42.242058 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:08:15.538249 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:15.538209 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:08:24.217118 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.217079 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n"] Jul 07 16:08:24.217588 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.217352 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41b83d67-29c9-4169-b70a-01cdb2264a19" containerName="authorino" Jul 07 16:08:24.217588 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.217363 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="41b83d67-29c9-4169-b70a-01cdb2264a19" containerName="authorino" Jul 07 16:08:24.217588 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.217408 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="41b83d67-29c9-4169-b70a-01cdb2264a19" containerName="authorino" Jul 07 16:08:24.220351 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.220323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.223190 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.223164 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-30f6fc42\"/\"openshift-service-ca.crt\"" Jul 07 16:08:24.224328 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.224309 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-30f6fc42\"/\"default-dockercfg-mqnh2\"" Jul 07 16:08:24.224429 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.224309 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-30f6fc42\"/\"kube-root-ca.crt\"" Jul 07 16:08:24.224429 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.224408 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-30f6fc42\"/\"e2e-lifecycle-model-30f6fc42-kserve-self-signed-certs\"" Jul 07 16:08:24.233038 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.233011 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n"] Jul 07 16:08:24.298295 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.298266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-home\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.298469 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.298301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b316c577-f89c-4938-b685-83d6c8929ff8-tls-certs\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.298469 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.298321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-tmp-dir\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.298469 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.298402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-dshm\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.298469 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.298447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-model-cache\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.298605 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.298473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7t2q\" (UniqueName: \"kubernetes.io/projected/b316c577-f89c-4938-b685-83d6c8929ff8-kube-api-access-q7t2q\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.398873 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.398834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-home\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.398873 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.398876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b316c577-f89c-4938-b685-83d6c8929ff8-tls-certs\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.399106 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.398907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-tmp-dir\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.399106 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.398955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-dshm\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.399106 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.399013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-model-cache\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.399106 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.399045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7t2q\" (UniqueName: \"kubernetes.io/projected/b316c577-f89c-4938-b685-83d6c8929ff8-kube-api-access-q7t2q\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.399334 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.399315 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-home\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.399381 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.399351 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-tmp-dir\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.399425 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.399387 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-model-cache\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.401420 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.401394 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-dshm\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.401580 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.401563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b316c577-f89c-4938-b685-83d6c8929ff8-tls-certs\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.407863 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.407841 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7t2q\" (UniqueName: \"kubernetes.io/projected/b316c577-f89c-4938-b685-83d6c8929ff8-kube-api-access-q7t2q\") pod \"e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.531723 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.531638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:24.680819 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.680790 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n"] Jul 07 16:08:24.681486 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:08:24.681458 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb316c577_f89c_4938_b685_83d6c8929ff8.slice/crio-6f5f887d4f0b17452de4317d60c659fd46a0bca999e11b4a70056fff90988de6 WatchSource:0}: Error finding container 6f5f887d4f0b17452de4317d60c659fd46a0bca999e11b4a70056fff90988de6: Status 404 returned error can't find the container with id 6f5f887d4f0b17452de4317d60c659fd46a0bca999e11b4a70056fff90988de6 Jul 07 16:08:24.996319 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.996278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" event={"ID":"b316c577-f89c-4938-b685-83d6c8929ff8","Type":"ContainerStarted","Data":"757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669"} Jul 07 16:08:24.996319 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:24.996326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" event={"ID":"b316c577-f89c-4938-b685-83d6c8929ff8","Type":"ContainerStarted","Data":"6f5f887d4f0b17452de4317d60c659fd46a0bca999e11b4a70056fff90988de6"} Jul 07 16:08:25.017182 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:25.017128 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" podStartSLOduration=1.017112538 podStartE2EDuration="1.017112538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:08:25.016188205 +0000 UTC m=+1674.615257488" watchObservedRunningTime="2026-07-07 16:08:25.017112538 +0000 UTC m=+1674.616181821" Jul 07 16:08:27.833198 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:27.833168 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:08:34.532453 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:34.532414 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:34.532945 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:34.532478 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:34.545188 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:34.545164 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:35.046421 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:35.046393 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:08:53.243466 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:08:53.243385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:09:04.942084 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:04.942047 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:09:06.912368 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:06.912333 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n"] Jul 07 16:09:21.933405 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:21.933335 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" podUID="b316c577-f89c-4938-b685-83d6c8929ff8" containerName="main" containerID="cri-o://757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669" gracePeriod=45 Jul 07 16:09:22.173942 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.173919 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:09:22.214202 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.214122 2576 generic.go:356] "Generic (PLEG): container finished" podID="b316c577-f89c-4938-b685-83d6c8929ff8" containerID="757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669" exitCode=0 Jul 07 16:09:22.214202 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.214169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" event={"ID":"b316c577-f89c-4938-b685-83d6c8929ff8","Type":"ContainerDied","Data":"757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669"} Jul 07 16:09:22.214202 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.214174 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" Jul 07 16:09:22.214439 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.214204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n" event={"ID":"b316c577-f89c-4938-b685-83d6c8929ff8","Type":"ContainerDied","Data":"6f5f887d4f0b17452de4317d60c659fd46a0bca999e11b4a70056fff90988de6"} Jul 07 16:09:22.214439 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.214222 2576 scope.go:117] "RemoveContainer" containerID="757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669" Jul 07 16:09:22.222708 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.222689 2576 scope.go:117] "RemoveContainer" containerID="757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669" Jul 07 16:09:22.223069 ip-10-0-137-175 kubenswrapper[2576]: E0707 16:09:22.223046 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669\": container with ID starting with 757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669 not found: ID does not exist" containerID="757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669" Jul 07 16:09:22.223137 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.223079 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669"} err="failed to get container status \"757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669\": rpc error: code = NotFound desc = could not find container \"757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669\": container with ID starting with 757354a21d84797c35dd5b93b1faff5d92a3019c10b39ba96eb86e7b3db41669 not found: ID does not exist" Jul 07 16:09:22.267107 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267073 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-dshm\") pod \"b316c577-f89c-4938-b685-83d6c8929ff8\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " Jul 07 16:09:22.267294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267121 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7t2q\" (UniqueName: \"kubernetes.io/projected/b316c577-f89c-4938-b685-83d6c8929ff8-kube-api-access-q7t2q\") pod \"b316c577-f89c-4938-b685-83d6c8929ff8\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " Jul 07 16:09:22.267294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-home\") pod \"b316c577-f89c-4938-b685-83d6c8929ff8\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " Jul 07 16:09:22.267294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267156 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-model-cache\") pod \"b316c577-f89c-4938-b685-83d6c8929ff8\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " Jul 07 16:09:22.267294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267176 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-tmp-dir\") pod \"b316c577-f89c-4938-b685-83d6c8929ff8\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " Jul 07 16:09:22.267294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267207 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b316c577-f89c-4938-b685-83d6c8929ff8-tls-certs\") pod \"b316c577-f89c-4938-b685-83d6c8929ff8\" (UID: \"b316c577-f89c-4938-b685-83d6c8929ff8\") " Jul 07 16:09:22.267541 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267417 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-model-cache" (OuterVolumeSpecName: "model-cache") pod "b316c577-f89c-4938-b685-83d6c8929ff8" (UID: "b316c577-f89c-4938-b685-83d6c8929ff8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:09:22.267541 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267431 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b316c577-f89c-4938-b685-83d6c8929ff8" (UID: "b316c577-f89c-4938-b685-83d6c8929ff8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:09:22.267541 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.267440 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-home" (OuterVolumeSpecName: "home") pod "b316c577-f89c-4938-b685-83d6c8929ff8" (UID: "b316c577-f89c-4938-b685-83d6c8929ff8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:09:22.269445 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.269413 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-dshm" (OuterVolumeSpecName: "dshm") pod "b316c577-f89c-4938-b685-83d6c8929ff8" (UID: "b316c577-f89c-4938-b685-83d6c8929ff8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:09:22.269556 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.269469 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b316c577-f89c-4938-b685-83d6c8929ff8-kube-api-access-q7t2q" (OuterVolumeSpecName: "kube-api-access-q7t2q") pod "b316c577-f89c-4938-b685-83d6c8929ff8" (UID: "b316c577-f89c-4938-b685-83d6c8929ff8"). InnerVolumeSpecName "kube-api-access-q7t2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:09:22.269556 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.269483 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b316c577-f89c-4938-b685-83d6c8929ff8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b316c577-f89c-4938-b685-83d6c8929ff8" (UID: "b316c577-f89c-4938-b685-83d6c8929ff8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:09:22.368789 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.368756 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7t2q\" (UniqueName: \"kubernetes.io/projected/b316c577-f89c-4938-b685-83d6c8929ff8-kube-api-access-q7t2q\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:09:22.368789 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.368784 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:09:22.368789 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.368794 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:09:22.369065 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.368803 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:09:22.369065 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.368811 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b316c577-f89c-4938-b685-83d6c8929ff8-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:09:22.369065 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.368819 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b316c577-f89c-4938-b685-83d6c8929ff8-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:09:22.535585 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.535559 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n"] Jul 07 16:09:22.540694 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.540672 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-30f6fc42/e2e-lifecycle-model-30f6fc42-kserve-6c48d74887-lr42n"] Jul 07 16:09:22.805811 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:22.805728 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b316c577-f89c-4938-b685-83d6c8929ff8" path="/var/lib/kubelet/pods/b316c577-f89c-4938-b685-83d6c8929ff8/volumes" Jul 07 16:09:46.644334 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:46.644298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:09:56.041862 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:09:56.041827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:10:07.842873 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:10:07.842835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:10:54.136686 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:10:54.136645 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:11:06.252098 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:06.252062 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:11:17.827290 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.827258 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs"] Jul 07 16:11:17.829541 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.827540 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b316c577-f89c-4938-b685-83d6c8929ff8" containerName="main" Jul 07 16:11:17.829541 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.827551 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b316c577-f89c-4938-b685-83d6c8929ff8" containerName="main" Jul 07 16:11:17.829541 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.827616 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b316c577-f89c-4938-b685-83d6c8929ff8" containerName="main" Jul 07 16:11:17.830377 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.830363 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.834310 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.834288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e6fe85\"/\"test-model-e6fe85-kserve-self-signed-certs\"" Jul 07 16:11:17.834310 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.834292 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e6fe85\"/\"openshift-service-ca.crt\"" Jul 07 16:11:17.834518 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.834339 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e6fe85\"/\"kube-root-ca.crt\"" Jul 07 16:11:17.834518 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.834390 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e6fe85\"/\"default-dockercfg-bsmw2\"" Jul 07 16:11:17.850176 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.850149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs"] Jul 07 16:11:17.882606 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.882565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-model-cache\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.882606 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.882606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-kube-api-access-6k8lk\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.882850 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.882642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-dshm\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.882850 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.882672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-home\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.882850 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.882700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tmp-dir\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.882850 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.882773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tls-certs\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.983462 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.983427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-dshm\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.983462 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.983466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-home\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.983733 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.983488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tmp-dir\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.983733 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.983535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tls-certs\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.983733 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.983575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-model-cache\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.983733 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.983606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-kube-api-access-6k8lk\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.984067 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.984042 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tmp-dir\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.984144 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.984058 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-home\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.984144 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.984098 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-model-cache\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.986144 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.986119 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-dshm\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.986334 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.986320 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tls-certs\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:17.991919 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:17.991902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-kube-api-access-6k8lk\") pod \"test-model-e6fe85-kserve-5b465fcf95-7w8qs\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:18.139216 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:18.139115 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:18.272218 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:18.272191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs"] Jul 07 16:11:18.274267 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:11:18.274238 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe9cf238_568e_4b7a_b4c0_7033be1ff3c0.slice/crio-f600856d530c264f1d98d9afef8ef07308b2a22df2cb1afb5e575429082d02b1 WatchSource:0}: Error finding container f600856d530c264f1d98d9afef8ef07308b2a22df2cb1afb5e575429082d02b1: Status 404 returned error can't find the container with id f600856d530c264f1d98d9afef8ef07308b2a22df2cb1afb5e575429082d02b1 Jul 07 16:11:18.634292 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:18.634252 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" event={"ID":"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0","Type":"ContainerStarted","Data":"b71a3cec86c5d873039a144cdd5c48ec87966821c5c8ca432235eb19a23a5a73"} Jul 07 16:11:18.634292 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:18.634293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" event={"ID":"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0","Type":"ContainerStarted","Data":"f600856d530c264f1d98d9afef8ef07308b2a22df2cb1afb5e575429082d02b1"} Jul 07 16:11:18.655271 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:18.655210 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" podStartSLOduration=1.6551901180000002 podStartE2EDuration="1.655190118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:11:18.653811754 +0000 UTC m=+1848.252881059" watchObservedRunningTime="2026-07-07 16:11:18.655190118 +0000 UTC m=+1848.254259406" Jul 07 16:11:20.942038 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:20.942000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:11:28.140331 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:28.140295 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:28.140760 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:28.140361 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:28.153360 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:28.153332 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:28.685431 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:28.685403 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:11:32.436925 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:32.436887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:11:40.518157 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.518126 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m"] Jul 07 16:11:40.521274 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.521258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.525387 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.525364 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-11d09f\"/\"default-dockercfg-qw7j8\"" Jul 07 16:11:40.525506 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.525474 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-11d09f\"/\"test-model-11d09f-kserve-self-signed-certs\"" Jul 07 16:11:40.525570 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.525528 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-11d09f\"/\"openshift-service-ca.crt\"" Jul 07 16:11:40.525628 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.525586 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-11d09f\"/\"kube-root-ca.crt\"" Jul 07 16:11:40.533104 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.533085 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m"] Jul 07 16:11:40.571602 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.571572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-model-cache\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.571602 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.571608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-home\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.571803 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.571628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-tmp-dir\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.571803 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.571645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a794d-e6d5-44fc-87af-ca4b069336ed-tls-certs\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.571803 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.571777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-dshm\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.572002 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.571821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqp4s\" (UniqueName: \"kubernetes.io/projected/a60a794d-e6d5-44fc-87af-ca4b069336ed-kube-api-access-pqp4s\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.672856 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.672823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqp4s\" (UniqueName: \"kubernetes.io/projected/a60a794d-e6d5-44fc-87af-ca4b069336ed-kube-api-access-pqp4s\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.673068 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.672879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-model-cache\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.673068 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.672900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-home\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.673068 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.672915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-tmp-dir\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.673068 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.672931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a794d-e6d5-44fc-87af-ca4b069336ed-tls-certs\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.673068 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.673023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-dshm\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.673316 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.673293 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-home\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.673390 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.673368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-model-cache\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.673448 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.673376 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-tmp-dir\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.675370 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.675343 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-dshm\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.675612 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.675595 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a794d-e6d5-44fc-87af-ca4b069336ed-tls-certs\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.682416 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.682396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqp4s\" (UniqueName: \"kubernetes.io/projected/a60a794d-e6d5-44fc-87af-ca4b069336ed-kube-api-access-pqp4s\") pod \"test-model-11d09f-kserve-785478ddf9-5vq5m\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.830331 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.830306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:40.963071 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:40.963043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m"] Jul 07 16:11:40.965633 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:11:40.965607 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda60a794d_e6d5_44fc_87af_ca4b069336ed.slice/crio-b08360a9c123bd868c37c166651e8d95aee7e5b5767dd984b323e06a992816de WatchSource:0}: Error finding container b08360a9c123bd868c37c166651e8d95aee7e5b5767dd984b323e06a992816de: Status 404 returned error can't find the container with id b08360a9c123bd868c37c166651e8d95aee7e5b5767dd984b323e06a992816de Jul 07 16:11:41.725385 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:41.725349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" event={"ID":"a60a794d-e6d5-44fc-87af-ca4b069336ed","Type":"ContainerStarted","Data":"9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff"} Jul 07 16:11:41.725751 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:41.725395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" event={"ID":"a60a794d-e6d5-44fc-87af-ca4b069336ed","Type":"ContainerStarted","Data":"b08360a9c123bd868c37c166651e8d95aee7e5b5767dd984b323e06a992816de"} Jul 07 16:11:41.748412 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:41.748366 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" podStartSLOduration=1.7483515710000002 podStartE2EDuration="1.748351571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:11:41.746549951 +0000 UTC m=+1871.345619245" watchObservedRunningTime="2026-07-07 16:11:41.748351571 +0000 UTC m=+1871.347420907" Jul 07 16:11:47.137350 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:47.137313 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:11:50.831054 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:50.831018 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:50.831476 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:50.831069 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:50.843750 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:50.843724 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:51.775583 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:51.775502 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:11:59.675235 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:11:59.675203 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs"] Jul 07 16:12:00.730124 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:00.730091 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m"] Jul 07 16:12:03.344393 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:03.344359 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:12:12.838864 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:12.838829 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:12:14.695286 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:14.695245 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" podUID="fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" containerName="main" containerID="cri-o://b71a3cec86c5d873039a144cdd5c48ec87966821c5c8ca432235eb19a23a5a73" gracePeriod=45 Jul 07 16:12:14.853446 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:14.853405 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" containerID="b71a3cec86c5d873039a144cdd5c48ec87966821c5c8ca432235eb19a23a5a73" exitCode=0 Jul 07 16:12:14.853600 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:14.853475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" event={"ID":"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0","Type":"ContainerDied","Data":"b71a3cec86c5d873039a144cdd5c48ec87966821c5c8ca432235eb19a23a5a73"} Jul 07 16:12:14.956286 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:14.956226 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:12:15.060506 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.060468 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-kube-api-access-6k8lk\") pod \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " Jul 07 16:12:15.060506 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.060509 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tmp-dir\") pod \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " Jul 07 16:12:15.060764 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.060527 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-home\") pod \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " Jul 07 16:12:15.060764 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.060559 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-dshm\") pod \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " Jul 07 16:12:15.060764 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.060582 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tls-certs\") pod \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " Jul 07 16:12:15.060764 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.060608 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-model-cache\") pod \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\" (UID: \"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0\") " Jul 07 16:12:15.060951 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.060841 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" (UID: "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:15.060951 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.060857 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-home" (OuterVolumeSpecName: "home") pod "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" (UID: "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:15.061055 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.061003 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" (UID: "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:15.062956 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.062923 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-kube-api-access-6k8lk" (OuterVolumeSpecName: "kube-api-access-6k8lk") pod "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" (UID: "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0"). InnerVolumeSpecName "kube-api-access-6k8lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:12:15.063112 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.062990 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-dshm" (OuterVolumeSpecName: "dshm") pod "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" (UID: "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:15.063112 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.063005 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" (UID: "fe9cf238-568e-4b7a-b4c0-7033be1ff3c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:12:15.162050 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.162015 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-kube-api-access-6k8lk\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:15.162050 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.162043 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:15.162050 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.162055 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:15.162294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.162063 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:15.162294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.162070 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:15.162294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.162078 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:15.750183 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.750139 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" podUID="a60a794d-e6d5-44fc-87af-ca4b069336ed" containerName="main" containerID="cri-o://9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff" gracePeriod=45 Jul 07 16:12:15.859342 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.859319 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" Jul 07 16:12:15.859342 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.859322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs" event={"ID":"fe9cf238-568e-4b7a-b4c0-7033be1ff3c0","Type":"ContainerDied","Data":"f600856d530c264f1d98d9afef8ef07308b2a22df2cb1afb5e575429082d02b1"} Jul 07 16:12:15.859562 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.859364 2576 scope.go:117] "RemoveContainer" containerID="b71a3cec86c5d873039a144cdd5c48ec87966821c5c8ca432235eb19a23a5a73" Jul 07 16:12:15.884609 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.884577 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs"] Jul 07 16:12:15.888642 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.888616 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e6fe85/test-model-e6fe85-kserve-5b465fcf95-7w8qs"] Jul 07 16:12:15.999367 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:15.999346 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:12:16.066755 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.066728 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-tmp-dir\") pod \"a60a794d-e6d5-44fc-87af-ca4b069336ed\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " Jul 07 16:12:16.066904 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.066785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-home\") pod \"a60a794d-e6d5-44fc-87af-ca4b069336ed\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " Jul 07 16:12:16.066904 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.066807 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqp4s\" (UniqueName: \"kubernetes.io/projected/a60a794d-e6d5-44fc-87af-ca4b069336ed-kube-api-access-pqp4s\") pod \"a60a794d-e6d5-44fc-87af-ca4b069336ed\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " Jul 07 16:12:16.066904 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.066828 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-model-cache\") pod \"a60a794d-e6d5-44fc-87af-ca4b069336ed\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " Jul 07 16:12:16.066904 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.066845 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-dshm\") pod \"a60a794d-e6d5-44fc-87af-ca4b069336ed\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " Jul 07 16:12:16.066904 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.066864 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a794d-e6d5-44fc-87af-ca4b069336ed-tls-certs\") pod \"a60a794d-e6d5-44fc-87af-ca4b069336ed\" (UID: \"a60a794d-e6d5-44fc-87af-ca4b069336ed\") " Jul 07 16:12:16.067183 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.067020 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a60a794d-e6d5-44fc-87af-ca4b069336ed" (UID: "a60a794d-e6d5-44fc-87af-ca4b069336ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:16.067183 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.067120 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:16.067183 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.067151 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-home" (OuterVolumeSpecName: "home") pod "a60a794d-e6d5-44fc-87af-ca4b069336ed" (UID: "a60a794d-e6d5-44fc-87af-ca4b069336ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:16.067183 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.067165 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "a60a794d-e6d5-44fc-87af-ca4b069336ed" (UID: "a60a794d-e6d5-44fc-87af-ca4b069336ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:16.069224 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.069193 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60a794d-e6d5-44fc-87af-ca4b069336ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a60a794d-e6d5-44fc-87af-ca4b069336ed" (UID: "a60a794d-e6d5-44fc-87af-ca4b069336ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:12:16.069224 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.069203 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-dshm" (OuterVolumeSpecName: "dshm") pod "a60a794d-e6d5-44fc-87af-ca4b069336ed" (UID: "a60a794d-e6d5-44fc-87af-ca4b069336ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:16.069379 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.069238 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a794d-e6d5-44fc-87af-ca4b069336ed-kube-api-access-pqp4s" (OuterVolumeSpecName: "kube-api-access-pqp4s") pod "a60a794d-e6d5-44fc-87af-ca4b069336ed" (UID: "a60a794d-e6d5-44fc-87af-ca4b069336ed"). InnerVolumeSpecName "kube-api-access-pqp4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:12:16.168116 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.168082 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:16.168116 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.168112 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqp4s\" (UniqueName: \"kubernetes.io/projected/a60a794d-e6d5-44fc-87af-ca4b069336ed-kube-api-access-pqp4s\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:16.168325 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.168128 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:16.168325 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.168141 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60a794d-e6d5-44fc-87af-ca4b069336ed-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:16.168325 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.168151 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60a794d-e6d5-44fc-87af-ca4b069336ed-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:12:16.805876 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.805837 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" path="/var/lib/kubelet/pods/fe9cf238-568e-4b7a-b4c0-7033be1ff3c0/volumes" Jul 07 16:12:16.864106 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.864068 2576 generic.go:356] "Generic (PLEG): container finished" podID="a60a794d-e6d5-44fc-87af-ca4b069336ed" containerID="9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff" exitCode=0 Jul 07 16:12:16.864106 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.864107 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" Jul 07 16:12:16.864313 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.864149 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" event={"ID":"a60a794d-e6d5-44fc-87af-ca4b069336ed","Type":"ContainerDied","Data":"9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff"} Jul 07 16:12:16.864313 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.864191 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m" event={"ID":"a60a794d-e6d5-44fc-87af-ca4b069336ed","Type":"ContainerDied","Data":"b08360a9c123bd868c37c166651e8d95aee7e5b5767dd984b323e06a992816de"} Jul 07 16:12:16.864313 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.864215 2576 scope.go:117] "RemoveContainer" containerID="9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff" Jul 07 16:12:16.872479 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.872462 2576 scope.go:117] "RemoveContainer" containerID="9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff" Jul 07 16:12:16.872748 ip-10-0-137-175 kubenswrapper[2576]: E0707 16:12:16.872726 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff\": container with ID starting with 9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff not found: ID does not exist" containerID="9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff" Jul 07 16:12:16.872794 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.872757 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff"} err="failed to get container status \"9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff\": rpc error: code = NotFound desc = could not find container \"9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff\": container with ID starting with 9fb4a5395244198a6282446f2c6ef06f8161268d01e868c166b136091a2dd8ff not found: ID does not exist" Jul 07 16:12:16.882337 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.882312 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m"] Jul 07 16:12:16.887143 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:16.887121 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-11d09f/test-model-11d09f-kserve-785478ddf9-5vq5m"] Jul 07 16:12:18.805879 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:18.805842 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60a794d-e6d5-44fc-87af-ca4b069336ed" path="/var/lib/kubelet/pods/a60a794d-e6d5-44fc-87af-ca4b069336ed/volumes" Jul 07 16:12:47.138322 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:12:47.138292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:13:01.945987 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:13:01.945931 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:13:16.629688 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:13:16.629653 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:13:25.744851 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:13:25.741042 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:13:39.130472 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:13:39.130439 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:13:51.148724 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:13:51.148692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:14:03.039759 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.039726 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn"] Jul 07 16:14:03.040271 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.040007 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60a794d-e6d5-44fc-87af-ca4b069336ed" containerName="main" Jul 07 16:14:03.040271 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.040019 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60a794d-e6d5-44fc-87af-ca4b069336ed" containerName="main" Jul 07 16:14:03.040271 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.040037 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" containerName="main" Jul 07 16:14:03.040271 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.040042 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" containerName="main" Jul 07 16:14:03.040271 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.040093 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe9cf238-568e-4b7a-b4c0-7033be1ff3c0" containerName="main" Jul 07 16:14:03.040271 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.040101 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60a794d-e6d5-44fc-87af-ca4b069336ed" containerName="main" Jul 07 16:14:03.043037 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.043020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.047082 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.047058 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"auth-test-model-12f2df-kserve-self-signed-certs\"" Jul 07 16:14:03.047211 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.047065 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"default-dockercfg-8fnvz\"" Jul 07 16:14:03.047211 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.047065 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"kube-root-ca.crt\"" Jul 07 16:14:03.047211 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.047105 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"openshift-service-ca.crt\"" Jul 07 16:14:03.053663 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.053642 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn"] Jul 07 16:14:03.212521 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.212477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-dshm\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.212521 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.212524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-home\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.212781 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.212550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-model-cache\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.212781 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.212649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-tmp-dir\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.212781 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.212707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbt46\" (UniqueName: \"kubernetes.io/projected/faf537e9-a43f-4218-857e-e1a5ce016af6-kube-api-access-rbt46\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.212781 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.212750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faf537e9-a43f-4218-857e-e1a5ce016af6-tls-certs\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314207 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-dshm\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314207 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-home\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314207 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-model-cache\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314468 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-tmp-dir\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314468 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbt46\" (UniqueName: \"kubernetes.io/projected/faf537e9-a43f-4218-857e-e1a5ce016af6-kube-api-access-rbt46\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314468 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faf537e9-a43f-4218-857e-e1a5ce016af6-tls-certs\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314680 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-model-cache\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314680 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314660 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-tmp-dir\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.314850 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.314705 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-home\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.316651 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.316631 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-dshm\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.316910 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.316892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faf537e9-a43f-4218-857e-e1a5ce016af6-tls-certs\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.323398 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.323377 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbt46\" (UniqueName: \"kubernetes.io/projected/faf537e9-a43f-4218-857e-e1a5ce016af6-kube-api-access-rbt46\") pod \"auth-test-model-12f2df-kserve-5cbd765d67-kxfsn\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.352103 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.352056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:03.487221 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:03.487189 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn"] Jul 07 16:14:03.490721 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:14:03.490693 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaf537e9_a43f_4218_857e_e1a5ce016af6.slice/crio-c7e045f9e1dbe288f7990143ae0c91dd66c98162cd5aa57c013523fa20a371a2 WatchSource:0}: Error finding container c7e045f9e1dbe288f7990143ae0c91dd66c98162cd5aa57c013523fa20a371a2: Status 404 returned error can't find the container with id c7e045f9e1dbe288f7990143ae0c91dd66c98162cd5aa57c013523fa20a371a2 Jul 07 16:14:04.261878 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:04.261847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" event={"ID":"faf537e9-a43f-4218-857e-e1a5ce016af6","Type":"ContainerStarted","Data":"cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7"} Jul 07 16:14:04.262315 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:04.261887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" event={"ID":"faf537e9-a43f-4218-857e-e1a5ce016af6","Type":"ContainerStarted","Data":"c7e045f9e1dbe288f7990143ae0c91dd66c98162cd5aa57c013523fa20a371a2"} Jul 07 16:14:04.281001 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:04.280934 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" podStartSLOduration=1.280917845 podStartE2EDuration="1.280917845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:14:04.280089208 +0000 UTC m=+2013.879158492" watchObservedRunningTime="2026-07-07 16:14:04.280917845 +0000 UTC m=+2013.879987129" Jul 07 16:14:06.248524 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:06.248489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:14:13.352553 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:13.352505 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:13.353085 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:13.352805 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:13.365157 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:13.365134 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:14.310010 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:14.309962 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:14:51.436207 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.436177 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b"] Jul 07 16:14:51.439382 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.439366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.442542 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.442512 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"kube-root-ca.crt\"" Jul 07 16:14:51.443955 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.443935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"default-dockercfg-chjsh\"" Jul 07 16:14:51.444085 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.443934 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"openshift-service-ca.crt\"" Jul 07 16:14:51.444085 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.444009 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"auth-test-model-ac6096-kserve-self-signed-certs\"" Jul 07 16:14:51.448964 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.448944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b"] Jul 07 16:14:51.500615 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.500541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-home\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.500615 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.500577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c140ea0-eb96-4a48-b470-e49722fc95d9-tls-certs\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.500833 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.500615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-dshm\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.500833 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.500666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-tmp-dir\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.500833 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.500700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p748x\" (UniqueName: \"kubernetes.io/projected/5c140ea0-eb96-4a48-b470-e49722fc95d9-kube-api-access-p748x\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.500833 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.500732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-model-cache\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.601459 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-home\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.601459 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c140ea0-eb96-4a48-b470-e49722fc95d9-tls-certs\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.601682 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-dshm\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.601682 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-tmp-dir\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.601682 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p748x\" (UniqueName: \"kubernetes.io/projected/5c140ea0-eb96-4a48-b470-e49722fc95d9-kube-api-access-p748x\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.601840 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-model-cache\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.601896 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601864 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-home\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.601947 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601894 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-tmp-dir\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.602040 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.601957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-model-cache\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.603924 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.603904 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-dshm\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.604189 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.604172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c140ea0-eb96-4a48-b470-e49722fc95d9-tls-certs\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.610381 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.610358 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p748x\" (UniqueName: \"kubernetes.io/projected/5c140ea0-eb96-4a48-b470-e49722fc95d9-kube-api-access-p748x\") pod \"auth-test-model-ac6096-kserve-69987b9575-rx26b\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.748800 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.748707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:14:51.882877 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:51.882846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b"] Jul 07 16:14:51.886131 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:14:51.886100 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c140ea0_eb96_4a48_b470_e49722fc95d9.slice/crio-7e4ef46cd99c500e851e350a7350ef509e9f3749ded6740b9f03299956fd7379 WatchSource:0}: Error finding container 7e4ef46cd99c500e851e350a7350ef509e9f3749ded6740b9f03299956fd7379: Status 404 returned error can't find the container with id 7e4ef46cd99c500e851e350a7350ef509e9f3749ded6740b9f03299956fd7379 Jul 07 16:14:52.448482 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:52.448442 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" event={"ID":"5c140ea0-eb96-4a48-b470-e49722fc95d9","Type":"ContainerStarted","Data":"5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec"} Jul 07 16:14:52.448482 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:52.448483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" event={"ID":"5c140ea0-eb96-4a48-b470-e49722fc95d9","Type":"ContainerStarted","Data":"7e4ef46cd99c500e851e350a7350ef509e9f3749ded6740b9f03299956fd7379"} Jul 07 16:14:52.467950 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:52.467897 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" podStartSLOduration=1.467880539 podStartE2EDuration="1.467880539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:14:52.466548483 +0000 UTC m=+2062.065617750" watchObservedRunningTime="2026-07-07 16:14:52.467880539 +0000 UTC m=+2062.066949822" Jul 07 16:14:55.237738 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:14:55.237701 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:15:01.749266 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:15:01.749208 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:15:01.749266 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:15:01.749287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:15:01.762100 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:15:01.762076 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:15:02.503802 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:15:02.503773 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:15:08.641342 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:15:08.641307 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:15:23.035195 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:15:23.035159 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:15:40.639172 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:15:40.639133 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:15:57.935788 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:15:57.935753 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:16:14.240769 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:14.240731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:16:29.934528 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:29.934449 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:16:40.033480 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:40.033446 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn"] Jul 07 16:16:40.853299 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:40.853266 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b"] Jul 07 16:16:44.533801 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:44.533768 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:16:55.053321 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.053283 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" podUID="faf537e9-a43f-4218-857e-e1a5ce016af6" containerName="main" containerID="cri-o://cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7" gracePeriod=45 Jul 07 16:16:55.244471 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.244439 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt"] Jul 07 16:16:55.247556 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.247538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.250261 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.250232 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"sub-test-model-12f2df-kserve-self-signed-certs\"" Jul 07 16:16:55.262602 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.262579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt"] Jul 07 16:16:55.297688 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.297657 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:16:55.334584 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.334553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tmp-dir\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.334735 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.334606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7r5p\" (UniqueName: \"kubernetes.io/projected/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-kube-api-access-z7r5p\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.334735 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.334630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-model-cache\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.334735 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.334704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-dshm\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.334855 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.334745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tls-certs\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.334855 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.334780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-home\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.435807 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.435773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faf537e9-a43f-4218-857e-e1a5ce016af6-tls-certs\") pod \"faf537e9-a43f-4218-857e-e1a5ce016af6\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " Jul 07 16:16:55.436019 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.435814 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-home\") pod \"faf537e9-a43f-4218-857e-e1a5ce016af6\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " Jul 07 16:16:55.436019 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.435861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-dshm\") pod \"faf537e9-a43f-4218-857e-e1a5ce016af6\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " Jul 07 16:16:55.436019 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.435907 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbt46\" (UniqueName: \"kubernetes.io/projected/faf537e9-a43f-4218-857e-e1a5ce016af6-kube-api-access-rbt46\") pod \"faf537e9-a43f-4218-857e-e1a5ce016af6\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " Jul 07 16:16:55.436019 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.435931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-tmp-dir\") pod \"faf537e9-a43f-4218-857e-e1a5ce016af6\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " Jul 07 16:16:55.436019 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.435989 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-model-cache\") pod \"faf537e9-a43f-4218-857e-e1a5ce016af6\" (UID: \"faf537e9-a43f-4218-857e-e1a5ce016af6\") " Jul 07 16:16:55.436275 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7r5p\" (UniqueName: \"kubernetes.io/projected/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-kube-api-access-z7r5p\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.436275 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-model-cache\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.436275 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436223 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-home" (OuterVolumeSpecName: "home") pod "faf537e9-a43f-4218-857e-e1a5ce016af6" (UID: "faf537e9-a43f-4218-857e-e1a5ce016af6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:55.436275 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-dshm\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.436469 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tls-certs\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.436469 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-home\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.436469 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436366 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-model-cache" (OuterVolumeSpecName: "model-cache") pod "faf537e9-a43f-4218-857e-e1a5ce016af6" (UID: "faf537e9-a43f-4218-857e-e1a5ce016af6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:55.436469 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tmp-dir\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.436685 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436509 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:55.436685 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436524 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:55.436685 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436505 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "faf537e9-a43f-4218-857e-e1a5ce016af6" (UID: "faf537e9-a43f-4218-857e-e1a5ce016af6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:55.436935 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.436808 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tmp-dir\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.437154 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.437092 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-model-cache\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.437290 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.437268 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-home\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.438453 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.438429 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf537e9-a43f-4218-857e-e1a5ce016af6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "faf537e9-a43f-4218-857e-e1a5ce016af6" (UID: "faf537e9-a43f-4218-857e-e1a5ce016af6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:16:55.438655 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.438625 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-dshm\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.438774 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.438694 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-dshm" (OuterVolumeSpecName: "dshm") pod "faf537e9-a43f-4218-857e-e1a5ce016af6" (UID: "faf537e9-a43f-4218-857e-e1a5ce016af6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:55.438774 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.438758 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf537e9-a43f-4218-857e-e1a5ce016af6-kube-api-access-rbt46" (OuterVolumeSpecName: "kube-api-access-rbt46") pod "faf537e9-a43f-4218-857e-e1a5ce016af6" (UID: "faf537e9-a43f-4218-857e-e1a5ce016af6"). InnerVolumeSpecName "kube-api-access-rbt46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:16:55.439011 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.438991 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tls-certs\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.447793 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.447772 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7r5p\" (UniqueName: \"kubernetes.io/projected/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-kube-api-access-z7r5p\") pod \"sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.536911 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.536876 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faf537e9-a43f-4218-857e-e1a5ce016af6-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:55.536911 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.536903 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:55.536911 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.536913 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbt46\" (UniqueName: \"kubernetes.io/projected/faf537e9-a43f-4218-857e-e1a5ce016af6-kube-api-access-rbt46\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:55.537163 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.536921 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faf537e9-a43f-4218-857e-e1a5ce016af6-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:55.556820 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.556793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:16:55.689956 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.689902 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt"] Jul 07 16:16:55.693576 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:16:55.693247 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5d6ad53_76c3_43b9_93c4_e5fa0b0f0ec7.slice/crio-5839f45587e7f5e92fcc12965881fad9c3a7342af69a8667a49a7be82735b527 WatchSource:0}: Error finding container 5839f45587e7f5e92fcc12965881fad9c3a7342af69a8667a49a7be82735b527: Status 404 returned error can't find the container with id 5839f45587e7f5e92fcc12965881fad9c3a7342af69a8667a49a7be82735b527 Jul 07 16:16:55.873338 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.873251 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" podUID="5c140ea0-eb96-4a48-b470-e49722fc95d9" containerName="main" containerID="cri-o://5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec" gracePeriod=45 Jul 07 16:16:55.932381 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.932350 2576 generic.go:356] "Generic (PLEG): container finished" podID="faf537e9-a43f-4218-857e-e1a5ce016af6" containerID="cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7" exitCode=0 Jul 07 16:16:55.932539 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.932396 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" Jul 07 16:16:55.932539 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.932437 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" event={"ID":"faf537e9-a43f-4218-857e-e1a5ce016af6","Type":"ContainerDied","Data":"cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7"} Jul 07 16:16:55.932539 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.932482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn" event={"ID":"faf537e9-a43f-4218-857e-e1a5ce016af6","Type":"ContainerDied","Data":"c7e045f9e1dbe288f7990143ae0c91dd66c98162cd5aa57c013523fa20a371a2"} Jul 07 16:16:55.932539 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.932501 2576 scope.go:117] "RemoveContainer" containerID="cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7" Jul 07 16:16:55.934039 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.934018 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" event={"ID":"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7","Type":"ContainerStarted","Data":"d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113"} Jul 07 16:16:55.934141 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.934052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" event={"ID":"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7","Type":"ContainerStarted","Data":"5839f45587e7f5e92fcc12965881fad9c3a7342af69a8667a49a7be82735b527"} Jul 07 16:16:55.955446 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.955411 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" podStartSLOduration=0.955397937 podStartE2EDuration="955.397937ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:16:55.953400716 +0000 UTC m=+2185.552470000" watchObservedRunningTime="2026-07-07 16:16:55.955397937 +0000 UTC m=+2185.554467221" Jul 07 16:16:55.965637 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.965610 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn"] Jul 07 16:16:55.968924 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.968903 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/auth-test-model-12f2df-kserve-5cbd765d67-kxfsn"] Jul 07 16:16:55.983505 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.983484 2576 scope.go:117] "RemoveContainer" containerID="cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7" Jul 07 16:16:55.983812 ip-10-0-137-175 kubenswrapper[2576]: E0707 16:16:55.983794 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7\": container with ID starting with cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7 not found: ID does not exist" containerID="cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7" Jul 07 16:16:55.983864 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:55.983821 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7"} err="failed to get container status \"cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7\": rpc error: code = NotFound desc = could not find container \"cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7\": container with ID starting with cdbdb438d753f95c6e7455e8fbe27edc3afa98a5e34da9192c973f05b308c1b7 not found: ID does not exist" Jul 07 16:16:56.123914 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.123855 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:16:56.240195 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240165 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-model-cache\") pod \"5c140ea0-eb96-4a48-b470-e49722fc95d9\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " Jul 07 16:16:56.240195 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240201 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p748x\" (UniqueName: \"kubernetes.io/projected/5c140ea0-eb96-4a48-b470-e49722fc95d9-kube-api-access-p748x\") pod \"5c140ea0-eb96-4a48-b470-e49722fc95d9\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " Jul 07 16:16:56.240399 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240248 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-tmp-dir\") pod \"5c140ea0-eb96-4a48-b470-e49722fc95d9\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " Jul 07 16:16:56.240399 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240275 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c140ea0-eb96-4a48-b470-e49722fc95d9-tls-certs\") pod \"5c140ea0-eb96-4a48-b470-e49722fc95d9\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " Jul 07 16:16:56.240399 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240298 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-dshm\") pod \"5c140ea0-eb96-4a48-b470-e49722fc95d9\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " Jul 07 16:16:56.240399 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240328 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-home\") pod \"5c140ea0-eb96-4a48-b470-e49722fc95d9\" (UID: \"5c140ea0-eb96-4a48-b470-e49722fc95d9\") " Jul 07 16:16:56.240598 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240444 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c140ea0-eb96-4a48-b470-e49722fc95d9" (UID: "5c140ea0-eb96-4a48-b470-e49722fc95d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:56.240598 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240532 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c140ea0-eb96-4a48-b470-e49722fc95d9" (UID: "5c140ea0-eb96-4a48-b470-e49722fc95d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:56.240703 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240622 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:56.240703 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240649 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:56.240703 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.240621 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-home" (OuterVolumeSpecName: "home") pod "5c140ea0-eb96-4a48-b470-e49722fc95d9" (UID: "5c140ea0-eb96-4a48-b470-e49722fc95d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:56.242680 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.242657 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c140ea0-eb96-4a48-b470-e49722fc95d9-kube-api-access-p748x" (OuterVolumeSpecName: "kube-api-access-p748x") pod "5c140ea0-eb96-4a48-b470-e49722fc95d9" (UID: "5c140ea0-eb96-4a48-b470-e49722fc95d9"). InnerVolumeSpecName "kube-api-access-p748x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:16:56.242757 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.242693 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c140ea0-eb96-4a48-b470-e49722fc95d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c140ea0-eb96-4a48-b470-e49722fc95d9" (UID: "5c140ea0-eb96-4a48-b470-e49722fc95d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:16:56.242757 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.242729 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-dshm" (OuterVolumeSpecName: "dshm") pod "5c140ea0-eb96-4a48-b470-e49722fc95d9" (UID: "5c140ea0-eb96-4a48-b470-e49722fc95d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:56.341208 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.341179 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c140ea0-eb96-4a48-b470-e49722fc95d9-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:56.341208 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.341202 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:56.341208 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.341210 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c140ea0-eb96-4a48-b470-e49722fc95d9-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:56.341208 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.341218 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p748x\" (UniqueName: \"kubernetes.io/projected/5c140ea0-eb96-4a48-b470-e49722fc95d9-kube-api-access-p748x\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:16:56.806261 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.806229 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faf537e9-a43f-4218-857e-e1a5ce016af6" path="/var/lib/kubelet/pods/faf537e9-a43f-4218-857e-e1a5ce016af6/volumes" Jul 07 16:16:56.940103 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.940070 2576 generic.go:356] "Generic (PLEG): container finished" podID="5c140ea0-eb96-4a48-b470-e49722fc95d9" containerID="5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec" exitCode=0 Jul 07 16:16:56.940302 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.940123 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" Jul 07 16:16:56.940302 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.940167 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" event={"ID":"5c140ea0-eb96-4a48-b470-e49722fc95d9","Type":"ContainerDied","Data":"5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec"} Jul 07 16:16:56.940302 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.940213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b" event={"ID":"5c140ea0-eb96-4a48-b470-e49722fc95d9","Type":"ContainerDied","Data":"7e4ef46cd99c500e851e350a7350ef509e9f3749ded6740b9f03299956fd7379"} Jul 07 16:16:56.940302 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.940242 2576 scope.go:117] "RemoveContainer" containerID="5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec" Jul 07 16:16:56.948777 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.948763 2576 scope.go:117] "RemoveContainer" containerID="5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec" Jul 07 16:16:56.949025 ip-10-0-137-175 kubenswrapper[2576]: E0707 16:16:56.949004 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec\": container with ID starting with 5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec not found: ID does not exist" containerID="5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec" Jul 07 16:16:56.949098 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.949034 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec"} err="failed to get container status \"5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec\": rpc error: code = NotFound desc = could not find container \"5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec\": container with ID starting with 5a839f93852317a1ea4eb07efa7f86d45474ecbad3dcf3202eb6cbc1f6effbec not found: ID does not exist" Jul 07 16:16:56.956620 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.956596 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b"] Jul 07 16:16:56.960736 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:56.960716 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/auth-test-model-ac6096-kserve-69987b9575-rx26b"] Jul 07 16:16:58.437778 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:58.437749 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:16:58.807727 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:16:58.807646 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c140ea0-eb96-4a48-b470-e49722fc95d9" path="/var/lib/kubelet/pods/5c140ea0-eb96-4a48-b470-e49722fc95d9/volumes" Jul 07 16:17:05.556950 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:05.556918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:17:05.557441 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:05.556988 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:17:05.569672 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:05.569648 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:17:05.990246 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:05.990220 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:17:12.836984 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.836931 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv"] Jul 07 16:17:12.837364 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.837208 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c140ea0-eb96-4a48-b470-e49722fc95d9" containerName="main" Jul 07 16:17:12.837364 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.837220 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c140ea0-eb96-4a48-b470-e49722fc95d9" containerName="main" Jul 07 16:17:12.837364 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.837230 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faf537e9-a43f-4218-857e-e1a5ce016af6" containerName="main" Jul 07 16:17:12.837364 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.837235 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="faf537e9-a43f-4218-857e-e1a5ce016af6" containerName="main" Jul 07 16:17:12.837364 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.837291 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c140ea0-eb96-4a48-b470-e49722fc95d9" containerName="main" Jul 07 16:17:12.837364 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.837302 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="faf537e9-a43f-4218-857e-e1a5ce016af6" containerName="main" Jul 07 16:17:12.839937 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.839922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.842846 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.842822 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"openshift-service-ca.crt\"" Jul 07 16:17:12.843007 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.842824 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"default-dockercfg-chjsh\"" Jul 07 16:17:12.844065 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.844046 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"sub-test-model-ac6096-kserve-self-signed-certs\"" Jul 07 16:17:12.844065 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.844052 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"kube-root-ca.crt\"" Jul 07 16:17:12.848480 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.848458 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv"] Jul 07 16:17:12.881405 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.881371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-tmp-dir\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.881575 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.881429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-home\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.881575 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.881457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrgl\" (UniqueName: \"kubernetes.io/projected/c22e5e38-228d-45b4-939c-d98226e39c0b-kube-api-access-qdrgl\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.881575 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.881492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-dshm\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.881674 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.881577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22e5e38-228d-45b4-939c-d98226e39c0b-tls-certs\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.881674 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.881621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-model-cache\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.982561 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.982518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22e5e38-228d-45b4-939c-d98226e39c0b-tls-certs\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.982738 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.982562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-model-cache\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.982738 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.982588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-tmp-dir\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.982738 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.982625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-home\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.982738 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.982643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrgl\" (UniqueName: \"kubernetes.io/projected/c22e5e38-228d-45b4-939c-d98226e39c0b-kube-api-access-qdrgl\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.982738 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.982691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-dshm\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.983118 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.983087 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-tmp-dir\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.983227 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.983157 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-home\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.983272 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.983246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-model-cache\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.985068 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.985040 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-dshm\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.985292 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.985278 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22e5e38-228d-45b4-939c-d98226e39c0b-tls-certs\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:12.991301 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:12.991279 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrgl\" (UniqueName: \"kubernetes.io/projected/c22e5e38-228d-45b4-939c-d98226e39c0b-kube-api-access-qdrgl\") pod \"sub-test-model-ac6096-kserve-99fc87cd5-79sbv\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:13.149256 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:13.149157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:13.283514 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:13.283489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv"] Jul 07 16:17:13.285622 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:17:13.285589 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc22e5e38_228d_45b4_939c_d98226e39c0b.slice/crio-0d36925598717e6dbcac1a88ee202b7208f7fc34ea08d996970c46f26045e613 WatchSource:0}: Error finding container 0d36925598717e6dbcac1a88ee202b7208f7fc34ea08d996970c46f26045e613: Status 404 returned error can't find the container with id 0d36925598717e6dbcac1a88ee202b7208f7fc34ea08d996970c46f26045e613 Jul 07 16:17:14.010137 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:14.010102 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" event={"ID":"c22e5e38-228d-45b4-939c-d98226e39c0b","Type":"ContainerStarted","Data":"9fbc551630a243ba21b2473b56788a90c110a436a03ca3d19d0c88b036cce3e3"} Jul 07 16:17:14.010137 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:14.010144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" event={"ID":"c22e5e38-228d-45b4-939c-d98226e39c0b","Type":"ContainerStarted","Data":"0d36925598717e6dbcac1a88ee202b7208f7fc34ea08d996970c46f26045e613"} Jul 07 16:17:14.029088 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:14.029032 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" podStartSLOduration=2.029014391 podStartE2EDuration="2.029014391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:17:14.028535527 +0000 UTC m=+2203.627604812" watchObservedRunningTime="2026-07-07 16:17:14.029014391 +0000 UTC m=+2203.628083675" Jul 07 16:17:19.235779 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:19.235735 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:17:23.150378 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:23.150328 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:23.150378 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:23.150393 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:23.163217 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:23.163193 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:24.060438 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:24.060408 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:17:32.632132 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:32.632092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:17:47.164039 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:47.164009 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt"] Jul 07 16:17:48.048607 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:48.048572 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv"] Jul 07 16:17:51.338270 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:17:51.338233 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:18:02.184312 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.184250 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" podUID="e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" containerName="main" containerID="cri-o://d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113" gracePeriod=45 Jul 07 16:18:02.428535 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.428513 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:18:02.501320 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501238 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7r5p\" (UniqueName: \"kubernetes.io/projected/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-kube-api-access-z7r5p\") pod \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " Jul 07 16:18:02.501320 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501282 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-dshm\") pod \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " Jul 07 16:18:02.501320 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501313 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-home\") pod \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " Jul 07 16:18:02.501524 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501338 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tls-certs\") pod \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " Jul 07 16:18:02.501524 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501479 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-model-cache\") pod \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " Jul 07 16:18:02.501628 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501538 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tmp-dir\") pod \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\" (UID: \"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7\") " Jul 07 16:18:02.501628 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501587 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-home" (OuterVolumeSpecName: "home") pod "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" (UID: "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:18:02.501761 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501716 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" (UID: "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:18:02.501822 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501785 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" (UID: "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:18:02.501822 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501816 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:02.501899 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.501837 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:02.503633 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.503614 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-dshm" (OuterVolumeSpecName: "dshm") pod "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" (UID: "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:18:02.504013 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.503994 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" (UID: "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:18:02.504074 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.504021 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-kube-api-access-z7r5p" (OuterVolumeSpecName: "kube-api-access-z7r5p") pod "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" (UID: "e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7"). InnerVolumeSpecName "kube-api-access-z7r5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:18:02.602479 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.602449 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7r5p\" (UniqueName: \"kubernetes.io/projected/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-kube-api-access-z7r5p\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:02.602479 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.602473 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:02.602479 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.602483 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:02.602675 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:02.602492 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:03.069116 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.069062 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" podUID="c22e5e38-228d-45b4-939c-d98226e39c0b" containerName="main" containerID="cri-o://9fbc551630a243ba21b2473b56788a90c110a436a03ca3d19d0c88b036cce3e3" gracePeriod=45 Jul 07 16:18:03.205075 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.205035 2576 generic.go:356] "Generic (PLEG): container finished" podID="e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" containerID="d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113" exitCode=0 Jul 07 16:18:03.205461 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.205082 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" Jul 07 16:18:03.205461 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.205129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" event={"ID":"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7","Type":"ContainerDied","Data":"d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113"} Jul 07 16:18:03.205461 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.205187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt" event={"ID":"e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7","Type":"ContainerDied","Data":"5839f45587e7f5e92fcc12965881fad9c3a7342af69a8667a49a7be82735b527"} Jul 07 16:18:03.205461 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.205214 2576 scope.go:117] "RemoveContainer" containerID="d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113" Jul 07 16:18:03.207294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.207245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" event={"ID":"c22e5e38-228d-45b4-939c-d98226e39c0b","Type":"ContainerDied","Data":"9fbc551630a243ba21b2473b56788a90c110a436a03ca3d19d0c88b036cce3e3"} Jul 07 16:18:03.207294 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.207137 2576 generic.go:356] "Generic (PLEG): container finished" podID="c22e5e38-228d-45b4-939c-d98226e39c0b" containerID="9fbc551630a243ba21b2473b56788a90c110a436a03ca3d19d0c88b036cce3e3" exitCode=0 Jul 07 16:18:03.213947 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.213930 2576 scope.go:117] "RemoveContainer" containerID="d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113" Jul 07 16:18:03.214235 ip-10-0-137-175 kubenswrapper[2576]: E0707 16:18:03.214219 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113\": container with ID starting with d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113 not found: ID does not exist" containerID="d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113" Jul 07 16:18:03.214286 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.214249 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113"} err="failed to get container status \"d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113\": rpc error: code = NotFound desc = could not find container \"d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113\": container with ID starting with d8690740cf9a4b0eff757732880079900e8a9e24eda1bab229a6e7116928e113 not found: ID does not exist" Jul 07 16:18:03.222256 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.222235 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt"] Jul 07 16:18:03.226453 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.226431 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/sub-test-model-12f2df-kserve-759c8fcc8f-fpbbt"] Jul 07 16:18:03.310211 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.310179 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:18:03.408802 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.408723 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-dshm\") pod \"c22e5e38-228d-45b4-939c-d98226e39c0b\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " Jul 07 16:18:03.408802 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.408777 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22e5e38-228d-45b4-939c-d98226e39c0b-tls-certs\") pod \"c22e5e38-228d-45b4-939c-d98226e39c0b\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " Jul 07 16:18:03.409036 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.408805 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-home\") pod \"c22e5e38-228d-45b4-939c-d98226e39c0b\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " Jul 07 16:18:03.409036 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.408826 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdrgl\" (UniqueName: \"kubernetes.io/projected/c22e5e38-228d-45b4-939c-d98226e39c0b-kube-api-access-qdrgl\") pod \"c22e5e38-228d-45b4-939c-d98226e39c0b\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " Jul 07 16:18:03.409036 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.408846 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-tmp-dir\") pod \"c22e5e38-228d-45b4-939c-d98226e39c0b\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " Jul 07 16:18:03.409036 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.408872 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-model-cache\") pod \"c22e5e38-228d-45b4-939c-d98226e39c0b\" (UID: \"c22e5e38-228d-45b4-939c-d98226e39c0b\") " Jul 07 16:18:03.409234 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.409149 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-home" (OuterVolumeSpecName: "home") pod "c22e5e38-228d-45b4-939c-d98226e39c0b" (UID: "c22e5e38-228d-45b4-939c-d98226e39c0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:18:03.409234 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.409166 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c22e5e38-228d-45b4-939c-d98226e39c0b" (UID: "c22e5e38-228d-45b4-939c-d98226e39c0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:18:03.409234 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.409215 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c22e5e38-228d-45b4-939c-d98226e39c0b" (UID: "c22e5e38-228d-45b4-939c-d98226e39c0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:18:03.411049 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.411024 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22e5e38-228d-45b4-939c-d98226e39c0b-kube-api-access-qdrgl" (OuterVolumeSpecName: "kube-api-access-qdrgl") pod "c22e5e38-228d-45b4-939c-d98226e39c0b" (UID: "c22e5e38-228d-45b4-939c-d98226e39c0b"). InnerVolumeSpecName "kube-api-access-qdrgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:18:03.411049 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.411037 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-dshm" (OuterVolumeSpecName: "dshm") pod "c22e5e38-228d-45b4-939c-d98226e39c0b" (UID: "c22e5e38-228d-45b4-939c-d98226e39c0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:18:03.411049 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.411043 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22e5e38-228d-45b4-939c-d98226e39c0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c22e5e38-228d-45b4-939c-d98226e39c0b" (UID: "c22e5e38-228d-45b4-939c-d98226e39c0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:18:03.509552 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.509521 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22e5e38-228d-45b4-939c-d98226e39c0b-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:03.509552 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.509546 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:03.509725 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.509569 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdrgl\" (UniqueName: \"kubernetes.io/projected/c22e5e38-228d-45b4-939c-d98226e39c0b-kube-api-access-qdrgl\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:03.509725 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.509578 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:03.509725 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.509588 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:03.509725 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:03.509595 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c22e5e38-228d-45b4-939c-d98226e39c0b-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:18:04.212799 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:04.212770 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" Jul 07 16:18:04.213313 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:04.212770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv" event={"ID":"c22e5e38-228d-45b4-939c-d98226e39c0b","Type":"ContainerDied","Data":"0d36925598717e6dbcac1a88ee202b7208f7fc34ea08d996970c46f26045e613"} Jul 07 16:18:04.213313 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:04.212905 2576 scope.go:117] "RemoveContainer" containerID="9fbc551630a243ba21b2473b56788a90c110a436a03ca3d19d0c88b036cce3e3" Jul 07 16:18:04.233501 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:04.233472 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv"] Jul 07 16:18:04.239604 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:04.239572 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/sub-test-model-ac6096-kserve-99fc87cd5-79sbv"] Jul 07 16:18:04.806679 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:04.806650 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22e5e38-228d-45b4-939c-d98226e39c0b" path="/var/lib/kubelet/pods/c22e5e38-228d-45b4-939c-d98226e39c0b/volumes" Jul 07 16:18:04.807042 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:04.807029 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" path="/var/lib/kubelet/pods/e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7/volumes" Jul 07 16:18:05.643226 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.643187 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts"] Jul 07 16:18:05.643632 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.643616 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" containerName="main" Jul 07 16:18:05.643676 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.643635 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" containerName="main" Jul 07 16:18:05.643676 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.643649 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22e5e38-228d-45b4-939c-d98226e39c0b" containerName="main" Jul 07 16:18:05.643676 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.643658 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22e5e38-228d-45b4-939c-d98226e39c0b" containerName="main" Jul 07 16:18:05.643764 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.643733 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22e5e38-228d-45b4-939c-d98226e39c0b" containerName="main" Jul 07 16:18:05.643764 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.643743 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d6ad53-76c3-43b9-93c4-e5fa0b0f0ec7" containerName="main" Jul 07 16:18:05.652263 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.652241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.656818 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.656795 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"kube-root-ca.crt\"" Jul 07 16:18:05.658242 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.658220 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"openshift-service-ca.crt\"" Jul 07 16:18:05.658372 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.658294 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"default-dockercfg-8fnvz\"" Jul 07 16:18:05.659210 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.658879 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-12f2df\"/\"rate-test-model-12f2df-kserve-self-signed-certs\"" Jul 07 16:18:05.660220 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.660200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts"] Jul 07 16:18:05.723926 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.723890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-model-cache\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.724119 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.723939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-home\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.724119 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.724014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tmp-dir\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.724119 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.724045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tls-certs\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.724231 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.724119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-dshm\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.724231 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.724162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nfb\" (UniqueName: \"kubernetes.io/projected/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-kube-api-access-l9nfb\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825139 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-home\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825139 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tmp-dir\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825416 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tls-certs\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825416 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-dshm\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825416 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfb\" (UniqueName: \"kubernetes.io/projected/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-kube-api-access-l9nfb\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825416 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-model-cache\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825624 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825509 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tmp-dir\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825624 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825568 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-home\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.825708 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.825642 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-model-cache\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.827673 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.827650 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-dshm\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.827776 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.827759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tls-certs\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.835737 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.835717 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nfb\" (UniqueName: \"kubernetes.io/projected/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-kube-api-access-l9nfb\") pod \"rate-test-model-12f2df-kserve-8469c877bc-8rnts\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:05.962200 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:05.962109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:06.092683 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:06.092651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts"] Jul 07 16:18:06.095358 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:18:06.095330 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2afb3b4_f9e7_4c47_87aa_ffec13073a69.slice/crio-295ec93dbb5e43f69a203ac6314a30910f6f2a2bf0172e1ca6a9283aea81a65d WatchSource:0}: Error finding container 295ec93dbb5e43f69a203ac6314a30910f6f2a2bf0172e1ca6a9283aea81a65d: Status 404 returned error can't find the container with id 295ec93dbb5e43f69a203ac6314a30910f6f2a2bf0172e1ca6a9283aea81a65d Jul 07 16:18:06.223232 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:06.223142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" event={"ID":"a2afb3b4-f9e7-4c47-87aa-ffec13073a69","Type":"ContainerStarted","Data":"dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da"} Jul 07 16:18:06.223232 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:06.223189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" event={"ID":"a2afb3b4-f9e7-4c47-87aa-ffec13073a69","Type":"ContainerStarted","Data":"295ec93dbb5e43f69a203ac6314a30910f6f2a2bf0172e1ca6a9283aea81a65d"} Jul 07 16:18:06.242741 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:06.242690 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" podStartSLOduration=1.242672906 podStartE2EDuration="1.242672906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:18:06.241143251 +0000 UTC m=+2255.840212560" watchObservedRunningTime="2026-07-07 16:18:06.242672906 +0000 UTC m=+2255.841742252" Jul 07 16:18:09.036632 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:09.036597 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:18:15.962355 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:15.962312 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:15.962763 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:15.962369 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:15.975468 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:15.975440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:16.282062 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:16.281985 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:18:21.631067 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.631031 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2"] Jul 07 16:18:21.634323 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.634306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.639078 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.639060 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"kube-root-ca.crt\"" Jul 07 16:18:21.640220 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.640198 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"openshift-service-ca.crt\"" Jul 07 16:18:21.640365 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.640341 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"rate-test-model-ac6096-kserve-self-signed-certs\"" Jul 07 16:18:21.640490 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.640208 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ac6096\"/\"default-dockercfg-chjsh\"" Jul 07 16:18:21.645218 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.645197 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2"] Jul 07 16:18:21.752126 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.752090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rsvq\" (UniqueName: \"kubernetes.io/projected/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-kube-api-access-4rsvq\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.752291 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.752132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-home\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.752291 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.752199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-dshm\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.752291 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.752219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-model-cache\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.752291 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.752264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tmp-dir\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.752426 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.752289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tls-certs\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.852983 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.852929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsvq\" (UniqueName: \"kubernetes.io/projected/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-kube-api-access-4rsvq\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.853161 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.852993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-home\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.853161 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.853031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-dshm\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.853161 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.853047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-model-cache\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.853161 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.853068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tmp-dir\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.853161 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.853088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tls-certs\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.853497 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.853473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-model-cache\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.853545 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.853487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-home\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.853586 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.853543 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tmp-dir\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.855426 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.855407 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-dshm\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.855774 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.855757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tls-certs\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.861761 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.861718 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rsvq\" (UniqueName: \"kubernetes.io/projected/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-kube-api-access-4rsvq\") pod \"rate-test-model-ac6096-kserve-6c8874c575-9cqs2\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:21.944807 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:21.944708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:22.073934 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:22.073881 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2"] Jul 07 16:18:22.076115 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:18:22.076092 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5a7f1f0_e3db_465b_84de_c6a4c197e18d.slice/crio-9ec466aed71e1a42c911497a55f64b28e703c88da11b1ea94f5cfeea03b0413c WatchSource:0}: Error finding container 9ec466aed71e1a42c911497a55f64b28e703c88da11b1ea94f5cfeea03b0413c: Status 404 returned error can't find the container with id 9ec466aed71e1a42c911497a55f64b28e703c88da11b1ea94f5cfeea03b0413c Jul 07 16:18:22.294373 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:22.294282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" event={"ID":"b5a7f1f0-e3db-465b-84de-c6a4c197e18d","Type":"ContainerStarted","Data":"7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c"} Jul 07 16:18:22.294373 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:22.294333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" event={"ID":"b5a7f1f0-e3db-465b-84de-c6a4c197e18d","Type":"ContainerStarted","Data":"9ec466aed71e1a42c911497a55f64b28e703c88da11b1ea94f5cfeea03b0413c"} Jul 07 16:18:22.312659 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:22.312597 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" podStartSLOduration=1.312581223 podStartE2EDuration="1.312581223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:18:22.311168225 +0000 UTC m=+2271.910237512" watchObservedRunningTime="2026-07-07 16:18:22.312581223 +0000 UTC m=+2271.911650543" Jul 07 16:18:26.336371 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:26.336337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:18:31.944837 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:31.944794 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:31.945269 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:31.944859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:31.957329 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:31.957302 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:32.343618 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:32.343588 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:18:55.340665 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:18:55.340630 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:19:09.233398 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:09.233362 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:19:14.953545 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:14.953512 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts"] Jul 07 16:19:15.830913 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:15.830874 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2"] Jul 07 16:19:26.144325 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:26.144245 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:19:29.973411 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:29.973376 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" podUID="a2afb3b4-f9e7-4c47-87aa-ffec13073a69" containerName="main" containerID="cri-o://dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da" gracePeriod=45 Jul 07 16:19:30.214400 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.214372 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:19:30.325473 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325446 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9nfb\" (UniqueName: \"kubernetes.io/projected/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-kube-api-access-l9nfb\") pod \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " Jul 07 16:19:30.325613 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325487 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-model-cache\") pod \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " Jul 07 16:19:30.325613 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325520 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-home\") pod \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " Jul 07 16:19:30.325613 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tmp-dir\") pod \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " Jul 07 16:19:30.325741 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325632 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-dshm\") pod \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " Jul 07 16:19:30.325741 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325667 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tls-certs\") pod \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\" (UID: \"a2afb3b4-f9e7-4c47-87aa-ffec13073a69\") " Jul 07 16:19:30.325836 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325802 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2afb3b4-f9e7-4c47-87aa-ffec13073a69" (UID: "a2afb3b4-f9e7-4c47-87aa-ffec13073a69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:19:30.325898 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325819 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-home" (OuterVolumeSpecName: "home") pod "a2afb3b4-f9e7-4c47-87aa-ffec13073a69" (UID: "a2afb3b4-f9e7-4c47-87aa-ffec13073a69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:19:30.325898 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325874 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2afb3b4-f9e7-4c47-87aa-ffec13073a69" (UID: "a2afb3b4-f9e7-4c47-87aa-ffec13073a69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:19:30.326039 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325934 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:30.326039 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325945 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:30.326039 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.325958 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:30.327870 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.327844 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-dshm" (OuterVolumeSpecName: "dshm") pod "a2afb3b4-f9e7-4c47-87aa-ffec13073a69" (UID: "a2afb3b4-f9e7-4c47-87aa-ffec13073a69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:19:30.327964 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.327926 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-kube-api-access-l9nfb" (OuterVolumeSpecName: "kube-api-access-l9nfb") pod "a2afb3b4-f9e7-4c47-87aa-ffec13073a69" (UID: "a2afb3b4-f9e7-4c47-87aa-ffec13073a69"). InnerVolumeSpecName "kube-api-access-l9nfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:19:30.327964 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.327935 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2afb3b4-f9e7-4c47-87aa-ffec13073a69" (UID: "a2afb3b4-f9e7-4c47-87aa-ffec13073a69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:19:30.432095 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.427609 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9nfb\" (UniqueName: \"kubernetes.io/projected/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-kube-api-access-l9nfb\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:30.432095 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.427665 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:30.432095 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.427756 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2afb3b4-f9e7-4c47-87aa-ffec13073a69-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:30.563519 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.563482 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2afb3b4-f9e7-4c47-87aa-ffec13073a69" containerID="dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da" exitCode=0 Jul 07 16:19:30.563687 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.563545 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" Jul 07 16:19:30.563687 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.563570 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" event={"ID":"a2afb3b4-f9e7-4c47-87aa-ffec13073a69","Type":"ContainerDied","Data":"dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da"} Jul 07 16:19:30.563687 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.563621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts" event={"ID":"a2afb3b4-f9e7-4c47-87aa-ffec13073a69","Type":"ContainerDied","Data":"295ec93dbb5e43f69a203ac6314a30910f6f2a2bf0172e1ca6a9283aea81a65d"} Jul 07 16:19:30.563687 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.563638 2576 scope.go:117] "RemoveContainer" containerID="dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da" Jul 07 16:19:30.572154 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.572137 2576 scope.go:117] "RemoveContainer" containerID="dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da" Jul 07 16:19:30.572437 ip-10-0-137-175 kubenswrapper[2576]: E0707 16:19:30.572418 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da\": container with ID starting with dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da not found: ID does not exist" containerID="dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da" Jul 07 16:19:30.572495 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.572447 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da"} err="failed to get container status \"dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da\": rpc error: code = NotFound desc = could not find container \"dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da\": container with ID starting with dcd1a6a207bb348ed51058d268e583731ab3374fec9be1cce4f0858adb8c75da not found: ID does not exist" Jul 07 16:19:30.583597 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.583535 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts"] Jul 07 16:19:30.590016 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.589990 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-12f2df/rate-test-model-12f2df-kserve-8469c877bc-8rnts"] Jul 07 16:19:30.806905 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.806870 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2afb3b4-f9e7-4c47-87aa-ffec13073a69" path="/var/lib/kubelet/pods/a2afb3b4-f9e7-4c47-87aa-ffec13073a69/volumes" Jul 07 16:19:30.850835 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:30.850749 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" podUID="b5a7f1f0-e3db-465b-84de-c6a4c197e18d" containerName="main" containerID="cri-o://7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c" gracePeriod=45 Jul 07 16:19:31.094364 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.094342 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:19:31.234849 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.234750 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tls-certs\") pod \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " Jul 07 16:19:31.234849 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.234796 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-home\") pod \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " Jul 07 16:19:31.234849 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.234815 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-model-cache\") pod \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " Jul 07 16:19:31.234849 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.234832 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tmp-dir\") pod \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " Jul 07 16:19:31.235300 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.234868 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-dshm\") pod \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " Jul 07 16:19:31.235300 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.234907 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rsvq\" (UniqueName: \"kubernetes.io/projected/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-kube-api-access-4rsvq\") pod \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\" (UID: \"b5a7f1f0-e3db-465b-84de-c6a4c197e18d\") " Jul 07 16:19:31.235300 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.235181 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5a7f1f0-e3db-465b-84de-c6a4c197e18d" (UID: "b5a7f1f0-e3db-465b-84de-c6a4c197e18d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:19:31.235300 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.235276 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-home" (OuterVolumeSpecName: "home") pod "b5a7f1f0-e3db-465b-84de-c6a4c197e18d" (UID: "b5a7f1f0-e3db-465b-84de-c6a4c197e18d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:19:31.235500 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.235444 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5a7f1f0-e3db-465b-84de-c6a4c197e18d" (UID: "b5a7f1f0-e3db-465b-84de-c6a4c197e18d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:19:31.237164 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.237137 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-dshm" (OuterVolumeSpecName: "dshm") pod "b5a7f1f0-e3db-465b-84de-c6a4c197e18d" (UID: "b5a7f1f0-e3db-465b-84de-c6a4c197e18d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:19:31.237221 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.237191 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-kube-api-access-4rsvq" (OuterVolumeSpecName: "kube-api-access-4rsvq") pod "b5a7f1f0-e3db-465b-84de-c6a4c197e18d" (UID: "b5a7f1f0-e3db-465b-84de-c6a4c197e18d"). InnerVolumeSpecName "kube-api-access-4rsvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:19:31.237264 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.237239 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5a7f1f0-e3db-465b-84de-c6a4c197e18d" (UID: "b5a7f1f0-e3db-465b-84de-c6a4c197e18d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:19:31.336119 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.336081 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tls-certs\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:31.336119 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.336111 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-home\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:31.336119 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.336123 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-model-cache\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:31.336359 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.336133 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-tmp-dir\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:31.336359 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.336143 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-dshm\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:31.336359 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.336154 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rsvq\" (UniqueName: \"kubernetes.io/projected/b5a7f1f0-e3db-465b-84de-c6a4c197e18d-kube-api-access-4rsvq\") on node \"ip-10-0-137-175.ec2.internal\" DevicePath \"\"" Jul 07 16:19:31.568104 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.568069 2576 generic.go:356] "Generic (PLEG): container finished" podID="b5a7f1f0-e3db-465b-84de-c6a4c197e18d" containerID="7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c" exitCode=0 Jul 07 16:19:31.568277 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.568111 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" Jul 07 16:19:31.568277 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.568154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" event={"ID":"b5a7f1f0-e3db-465b-84de-c6a4c197e18d","Type":"ContainerDied","Data":"7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c"} Jul 07 16:19:31.568277 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.568198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2" event={"ID":"b5a7f1f0-e3db-465b-84de-c6a4c197e18d","Type":"ContainerDied","Data":"9ec466aed71e1a42c911497a55f64b28e703c88da11b1ea94f5cfeea03b0413c"} Jul 07 16:19:31.568277 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.568219 2576 scope.go:117] "RemoveContainer" containerID="7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c" Jul 07 16:19:31.577319 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.577304 2576 scope.go:117] "RemoveContainer" containerID="7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c" Jul 07 16:19:31.577588 ip-10-0-137-175 kubenswrapper[2576]: E0707 16:19:31.577568 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c\": container with ID starting with 7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c not found: ID does not exist" containerID="7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c" Jul 07 16:19:31.577641 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.577598 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c"} err="failed to get container status \"7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c\": rpc error: code = NotFound desc = could not find container \"7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c\": container with ID starting with 7ea81a312afd73bf074e9a4aa9e893ec08b63ee13f761efa036666916d7b505c not found: ID does not exist" Jul 07 16:19:31.589422 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.589400 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2"] Jul 07 16:19:31.592952 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:31.592930 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ac6096/rate-test-model-ac6096-kserve-6c8874c575-9cqs2"] Jul 07 16:19:32.806405 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:19:32.806372 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a7f1f0-e3db-465b-84de-c6a4c197e18d" path="/var/lib/kubelet/pods/b5a7f1f0-e3db-465b-84de-c6a4c197e18d/volumes" Jul 07 16:20:03.631060 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:03.631018 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-p89nr"] Jul 07 16:20:07.347839 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:07.347805 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-798669dbf4-25m9v_0a2f9f6b-a0ff-44f7-89ad-b441384f1688/authorino/0.log" Jul 07 16:20:12.995282 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:12.995250 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cpd72_28547c6f-99be-4a11-9257-fda364906d05/manager/0.log" Jul 07 16:20:13.236340 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:13.236311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77db849c7c-76jc9_787274ec-e837-43e3-bf5d-d246be0b8711/copy-manifests/0.log" Jul 07 16:20:13.249211 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:13.249140 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77db849c7c-76jc9_787274ec-e837-43e3-bf5d-d246be0b8711/manager/0.log" Jul 07 16:20:14.785025 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:14.784995 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-798669dbf4-25m9v_0a2f9f6b-a0ff-44f7-89ad-b441384f1688/authorino/0.log" Jul 07 16:20:15.493076 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:15.493043 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-p89nr_cfca23ef-78fc-4d21-80f4-17d477df477e/limitador/0.log" Jul 07 16:20:16.226948 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:16.226923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6669df4868-vm26x_7a1f55ef-5235-42be-872b-483bc1f97143/kube-auth-proxy/0.log" Jul 07 16:20:17.784144 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:17.784116 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-fjbbs_5e7616e2-b9a8-4858-b83c-4c99e24b3b72/main/0.log" Jul 07 16:20:22.162191 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.162155 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lhshj/must-gather-8j2hz"] Jul 07 16:20:22.162642 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.162415 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a7f1f0-e3db-465b-84de-c6a4c197e18d" containerName="main" Jul 07 16:20:22.162642 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.162425 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a7f1f0-e3db-465b-84de-c6a4c197e18d" containerName="main" Jul 07 16:20:22.162642 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.162436 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2afb3b4-f9e7-4c47-87aa-ffec13073a69" containerName="main" Jul 07 16:20:22.162642 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.162441 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2afb3b4-f9e7-4c47-87aa-ffec13073a69" containerName="main" Jul 07 16:20:22.162642 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.162497 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2afb3b4-f9e7-4c47-87aa-ffec13073a69" containerName="main" Jul 07 16:20:22.162642 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.162506 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a7f1f0-e3db-465b-84de-c6a4c197e18d" containerName="main" Jul 07 16:20:22.165382 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.165365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lhshj/must-gather-8j2hz" Jul 07 16:20:22.168557 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.168532 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lhshj\"/\"default-dockercfg-wfbq2\"" Jul 07 16:20:22.168696 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.168532 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lhshj\"/\"openshift-service-ca.crt\"" Jul 07 16:20:22.168696 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.168566 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lhshj\"/\"kube-root-ca.crt\"" Jul 07 16:20:22.175916 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.175896 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lhshj/must-gather-8j2hz"] Jul 07 16:20:22.223378 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.223322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abbf37f5-7bbe-4be5-b42f-d2932819f9e5-must-gather-output\") pod \"must-gather-8j2hz\" (UID: \"abbf37f5-7bbe-4be5-b42f-d2932819f9e5\") " pod="openshift-must-gather-lhshj/must-gather-8j2hz" Jul 07 16:20:22.223550 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.223422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jxd\" (UniqueName: \"kubernetes.io/projected/abbf37f5-7bbe-4be5-b42f-d2932819f9e5-kube-api-access-h4jxd\") pod \"must-gather-8j2hz\" (UID: \"abbf37f5-7bbe-4be5-b42f-d2932819f9e5\") " pod="openshift-must-gather-lhshj/must-gather-8j2hz" Jul 07 16:20:22.324693 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.324658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abbf37f5-7bbe-4be5-b42f-d2932819f9e5-must-gather-output\") pod \"must-gather-8j2hz\" (UID: \"abbf37f5-7bbe-4be5-b42f-d2932819f9e5\") " pod="openshift-must-gather-lhshj/must-gather-8j2hz" Jul 07 16:20:22.324850 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.324706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jxd\" (UniqueName: \"kubernetes.io/projected/abbf37f5-7bbe-4be5-b42f-d2932819f9e5-kube-api-access-h4jxd\") pod \"must-gather-8j2hz\" (UID: \"abbf37f5-7bbe-4be5-b42f-d2932819f9e5\") " pod="openshift-must-gather-lhshj/must-gather-8j2hz" Jul 07 16:20:22.325041 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.325022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abbf37f5-7bbe-4be5-b42f-d2932819f9e5-must-gather-output\") pod \"must-gather-8j2hz\" (UID: \"abbf37f5-7bbe-4be5-b42f-d2932819f9e5\") " pod="openshift-must-gather-lhshj/must-gather-8j2hz" Jul 07 16:20:22.333349 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.333326 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jxd\" (UniqueName: \"kubernetes.io/projected/abbf37f5-7bbe-4be5-b42f-d2932819f9e5-kube-api-access-h4jxd\") pod \"must-gather-8j2hz\" (UID: \"abbf37f5-7bbe-4be5-b42f-d2932819f9e5\") " pod="openshift-must-gather-lhshj/must-gather-8j2hz" Jul 07 16:20:22.474228 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.474134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lhshj/must-gather-8j2hz" Jul 07 16:20:22.604076 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.604041 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lhshj/must-gather-8j2hz"] Jul 07 16:20:22.606923 ip-10-0-137-175 kubenswrapper[2576]: W0707 16:20:22.606893 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabbf37f5_7bbe_4be5_b42f_d2932819f9e5.slice/crio-fb60d77dc09e24cf88e7393a065c9c72c93a7b156f5010f991dfc86b5ff9d9e5 WatchSource:0}: Error finding container fb60d77dc09e24cf88e7393a065c9c72c93a7b156f5010f991dfc86b5ff9d9e5: Status 404 returned error can't find the container with id fb60d77dc09e24cf88e7393a065c9c72c93a7b156f5010f991dfc86b5ff9d9e5 Jul 07 16:20:22.608860 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.608842 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:20:22.769017 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:22.768907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lhshj/must-gather-8j2hz" event={"ID":"abbf37f5-7bbe-4be5-b42f-d2932819f9e5","Type":"ContainerStarted","Data":"fb60d77dc09e24cf88e7393a065c9c72c93a7b156f5010f991dfc86b5ff9d9e5"} Jul 07 16:20:23.777145 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:23.777037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lhshj/must-gather-8j2hz" event={"ID":"abbf37f5-7bbe-4be5-b42f-d2932819f9e5","Type":"ContainerStarted","Data":"2e524b46e7e5d50e1d028639b177c47c5a930e826f49e138fc519c439441f25f"} Jul 07 16:20:23.777145 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:23.777101 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lhshj/must-gather-8j2hz" event={"ID":"abbf37f5-7bbe-4be5-b42f-d2932819f9e5","Type":"ContainerStarted","Data":"4672c3b31779611458349b0d62f7c7941c04830d0494c218f151d1af187f7869"} Jul 07 16:20:23.796230 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:23.795908 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lhshj/must-gather-8j2hz" podStartSLOduration=1.056588789 podStartE2EDuration="1.795882078s" totalImagesPullingTime="739.293289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:20:23.794695471 +0000 UTC m=+2393.393764755" watchObservedRunningTime="2026-07-07 16:20:23.795882078 +0000 UTC m=+2393.394951367" Jul 07 16:20:25.015288 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:25.015256 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cvlj7_27cbb2a7-11c0-43e1-9d8f-1f7e630465d1/global-pull-secret-syncer/0.log" Jul 07 16:20:25.197008 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:25.196957 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xfwf2_791f6cb8-9518-45c6-aad0-aa14c4271724/konnectivity-agent/0.log" Jul 07 16:20:25.219899 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:25.219869 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-175.ec2.internal_ed097c754df8ae0ea286f9bc978e4472/haproxy/0.log" Jul 07 16:20:30.633958 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:30.633915 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-798669dbf4-25m9v_0a2f9f6b-a0ff-44f7-89ad-b441384f1688/authorino/0.log" Jul 07 16:20:30.961362 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:30.961291 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-p89nr_cfca23ef-78fc-4d21-80f4-17d477df477e/limitador/0.log" Jul 07 16:20:33.005917 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:33.005880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dcsnk_a1861a98-2c66-4753-a243-15132f84c77b/node-exporter/0.log" Jul 07 16:20:33.030448 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:33.030412 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dcsnk_a1861a98-2c66-4753-a243-15132f84c77b/kube-rbac-proxy/0.log" Jul 07 16:20:33.058458 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:33.058429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dcsnk_a1861a98-2c66-4753-a243-15132f84c77b/init-textfile/0.log" Jul 07 16:20:34.691555 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.691506 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7"] Jul 07 16:20:34.697192 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.697164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.708174 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.708149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7"] Jul 07 16:20:34.754526 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.754485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-sys\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.754760 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.754741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6kw\" (UniqueName: \"kubernetes.io/projected/56caeb60-2f5d-4c91-a2dd-a524b0435688-kube-api-access-rp6kw\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.754919 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.754903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-podres\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.755043 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.755030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-proc\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.755158 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.755145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-lib-modules\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856191 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-sys\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856377 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6kw\" (UniqueName: \"kubernetes.io/projected/56caeb60-2f5d-4c91-a2dd-a524b0435688-kube-api-access-rp6kw\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856377 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856236 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-sys\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856377 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-podres\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856377 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-proc\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856377 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-lib-modules\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856377 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856370 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-proc\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856689 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856425 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-podres\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.856689 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.856441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56caeb60-2f5d-4c91-a2dd-a524b0435688-lib-modules\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:34.867453 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:34.867414 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6kw\" (UniqueName: \"kubernetes.io/projected/56caeb60-2f5d-4c91-a2dd-a524b0435688-kube-api-access-rp6kw\") pod \"perf-node-gather-daemonset-cjpb7\" (UID: \"56caeb60-2f5d-4c91-a2dd-a524b0435688\") " pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:35.009755 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:35.009668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:35.191714 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:35.189454 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7"] Jul 07 16:20:35.852008 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:35.851861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" event={"ID":"56caeb60-2f5d-4c91-a2dd-a524b0435688","Type":"ContainerStarted","Data":"0fd998d609990073126b54b49450fb5c359dbbd02a682f9b3e3991afc2f3fa58"} Jul 07 16:20:35.853008 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:35.852525 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" event={"ID":"56caeb60-2f5d-4c91-a2dd-a524b0435688","Type":"ContainerStarted","Data":"12c83dd5a8dcb89a29588d0e6bf21c0846fba45767cd67397a7ab19b4e51d295"} Jul 07 16:20:35.853233 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:35.853201 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:35.873366 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:35.873302 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" podStartSLOduration=1.8732803599999999 podStartE2EDuration="1.87328036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:20:35.871768731 +0000 UTC m=+2405.470838015" watchObservedRunningTime="2026-07-07 16:20:35.87328036 +0000 UTC m=+2405.472349645" Jul 07 16:20:37.970200 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:37.970147 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tg7zm_d079eb86-acc0-4b1d-9cb5-873b28271068/dns/0.log" Jul 07 16:20:37.995371 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:37.995319 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tg7zm_d079eb86-acc0-4b1d-9cb5-873b28271068/kube-rbac-proxy/0.log" Jul 07 16:20:38.068415 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:38.068386 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mjdkq_63388bc4-351a-4315-b24f-53c259ab2b4b/dns-node-resolver/0.log" Jul 07 16:20:38.620697 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:38.620661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b8b55b46c-5cpb4_3258b7f5-c7b1-45df-94b7-da0f8af0e645/registry/0.log" Jul 07 16:20:38.642819 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:38.642790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9grq8_0f7b920c-de3b-4788-81cf-17b3640ed801/node-ca/0.log" Jul 07 16:20:39.861534 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:39.861505 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6669df4868-vm26x_7a1f55ef-5235-42be-872b-483bc1f97143/kube-auth-proxy/0.log" Jul 07 16:20:40.877986 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:40.877936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w6j8h_f5155845-d5c0-44b1-a045-9ae36b9bbfb9/serve-healthcheck-canary/0.log" Jul 07 16:20:41.380950 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:41.380922 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4b46w_dc0f2445-5253-4dec-abeb-f60ea932d598/kube-rbac-proxy/0.log" Jul 07 16:20:41.404960 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:41.404930 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4b46w_dc0f2445-5253-4dec-abeb-f60ea932d598/exporter/0.log" Jul 07 16:20:41.431944 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:41.431914 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4b46w_dc0f2445-5253-4dec-abeb-f60ea932d598/extractor/0.log" Jul 07 16:20:41.869825 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:41.869797 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lhshj/perf-node-gather-daemonset-cjpb7" Jul 07 16:20:44.541124 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:44.540988 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cpd72_28547c6f-99be-4a11-9257-fda364906d05/manager/0.log" Jul 07 16:20:44.727933 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:44.727871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77db849c7c-76jc9_787274ec-e837-43e3-bf5d-d246be0b8711/manager/0.log" Jul 07 16:20:44.763519 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:44.763491 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77db849c7c-76jc9_787274ec-e837-43e3-bf5d-d246be0b8711/copy-manifests/0.log" Jul 07 16:20:45.859886 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:45.859850 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6669df4868-vm26x_7a1f55ef-5235-42be-872b-483bc1f97143/kube-auth-proxy/0.log" Jul 07 16:20:46.152214 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:46.152157 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-778944b866-zlbn9_fb4e4aa4-dca6-4e0e-b575-3f74432b8c31/manager/0.log" Jul 07 16:20:47.457331 ip-10-0-137-175 kubenswrapper[2576]: I0707 16:20:47.457301 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6669df4868-vm26x_7a1f55ef-5235-42be-872b-483bc1f97143/kube-auth-proxy/0.log"